Methods and systems using fractional rank precision and mean average precision as test-retest reliability measures

ABSTRACT

Disclosed herein are methods and systems of evaluating test-retest precision using fractional rank precision or mean-average precision, comprising: a) collecting a test and a retest result of each subject, wherein the results are described in feature space(s) and collected from a vision test machine; b) selecting, a first test result of a first subject; c) calculating distances from the first test result to the retest result of each subject; d) assessing, a similarity between the first test result and the retest result of each subject by ranking the distances in a non-descending order; e) assessing a rank precision for the first subject based on a rank of a distance from the first test result to the retest result of the first subject; f) repeating b), c), d), and e) for each subject; and evaluating, the test-retest precision based on the rank precision for each of the plurality of subjects.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application is a continuation of U.S. application Ser. No.15/965,614, filed Apr. 27, 2018, which is a continuation of U.S.application Ser. No. 15/146,632, filed May 4, 2016, which claims thebenefit of U.S. Provisional Application No. 62/156,816 filed May 4,2015, the entire contents of which are incorporated herein by reference.

BACKGROUND

Clinically test-retest is typically used in monitoring of diseaseprogression and/or effects of therapeutic treatment. Test-retestreliability is indicated by the variation in measurement results takenon a same subject in a test and in a subsequent retest. Such variabilitycan be caused by a variety of factors including intra-individualvariability and/or variability in measurement devices. A measurement maybe considered as reliable when this variation is smaller than apre-determined acceptance threshold.

SUMMARY OF THE INVENTION

Vision is one of the most important senses for many everyday tasks, andblindness ranks highly among most-feared ailments. Therefore, earlydiagnosis and treatment of vision loss are critical. As a result,regular monitoring of visual function at least in at-risk populationswould be desirable. Certain vision tests rest on computationallyintensive algorithms to provide a more comprehensive description ofvisual function. Thus, they have the potential to improve clinical careand research by more precise measurement of the effects of diseaseprogression or ophthalmic interventions. A common proxy to studyprecision of a vision test is to assess its test-retest reliability;however, standard methods to assess clinical precision and test-retestreliability may be inadequate or even misleading for the more complex,higher-dimensional test outputs. Two major sources of imprecisiontypically are noise in the measurement device and moment-to-momentvariability of the physiological phenomenon under observation. Toestimate the contribution of noise in the device, precision is oftenassessed by the reliability of repeated measurements. The standard toolsfor this assessment are the intra-class correlation coefficient (ICC)and the Bland-Altman coefficient of repeatability (CoR). However, it isimportant to note that repeatability measures are an indirect assessmentonly; in the extreme case, a test with a binary outcome, as anon-limiting example, light perception, have almost perfect reliabilitybut little discriminatory power for most of the population. Furthermore,the ICC is dominated by the values at either end of the test range andis therefore sensitive to outliers. The Bland-Altman coefficient ofrepeatability, which is defined as 1.96 times the standard deviation ofdifferences between repeated measurements, does not suffer from thisproblem, and also may provide an intuitive threshold for how much changebetween two tests should be considered statistically significant.However, this threshold rests on the assumption that tests arehomoscedastic, as a non-limiting example, the measurement error isindependent of the magnitude of the ground truth (e.g. patients withpoor vision perform tests as reliably as normal-sighted controls); itsusefulness is limited by quantization of many tests, in addition to theevidence of heteroscedasticity in vision testing data. Moreover,absolute CoR values do not directly relate to clinical meaningfulness,and do not allow comparing the reliability of different tests withoutputs of different magnitude or different dimensionality.

The methods and systems disclosed herein, in various embodiments, adaptat least one method in machine learning and/or information retrievalfield and apply it in the measurement of similarity in test-retest pairsof at least one subject. In some embodiments, the methods and systemsdisclosed herein include at least a mean average precision method (MAP)adapted from machine learning and/or information retrieval. In someembodiments, the methods and systems disclosed herein include at least afractional ranking precision (FRP) method derived from machine learningand/or information retrieval field. The methods and systems disclosedherein, for non-limiting examples, MAP and FRP, improve the limitationsof traditional methods for the measurement of test-retest reliabilityand precision. Additionally, the methods and systems disclosed hereinprovide sensitivity (detection of subtle changes) and robustness (in thepresence of artifacts) in evaluating the effectiveness of vision-basedfeatures in the detection of critical vision changes caused by diseaseprogression and/or therapeutic interventions. Furthermore, the methodsand systems disclosed herein enable assessment of vision-based featuresin multi-dimensional feature space.

In one aspect, disclosed herein are computer-implemented methods ofevaluating test-retest precision of a test and a retest of a pluralityof subjects using fractional rank precision (FRP) or mean-averageprecision (MAP), comprising: a) collecting, by a computer, a test resultand a retest result of each of the plurality of subjects, wherein thetest result and the retest result are described in one or more featurespaces and one of the test result and the retest result is collectedfrom a vision test machine; b) selecting, by the computer, a firstsubject from the plurality of subjects and a first test result of thefirst subject; c) calculating, by the computer, distances from the firsttest result to the retest result of each of the plurality of subjects;d) assessing, by the computer, a similarity between the first testresult and the retest result of each of the plurality of subjects byranking the distances in a non-descending order; e) assessing, by thecomputer, a rank precision for the first subject based on a rank of adistance from the first test result to the retest result of the firstsubject; f) repeating, by the computer, steps b), c), d), and e) foreach of the plurality of subjects; and g) evaluating, by the computer,the test-retest precision based on the rank precision for each of theplurality of subjects. In some embodiments, the test is a first visiontest. In some embodiments, the retest is the first vision test or asecond vision test. In some embodiments, the first vision test or thesecond vision test is one or more selected from: a vision acuity test, aCSF test, and an OCT test. In some embodiments, the feature space isone-dimensional or multi-dimensional. In some embodiments, the featurespace comprises a feature. In some embodiments, the feature includes oneor more features selected from: a median AULCSF computed over thespatial frequency range of 1.5 to 6 cpd, a median AULCSF computed overthe spatial frequency range of 6 to 12 cpd, a median AULCSF computedover the spatial frequency range of 12 to 18 cpd, a median AULCSFcomputed over the spatial frequency range of 1.5 to 18 cpd, a CSFacuity, a parameter of CSF, a contrast sensitivity for at least onespatial frequency selected from 1, 1.5, 3, 6, 12, and 18 cpd, a peaksensitivity of the CSF, and a spatial frequency at which a CSF reaches apre-determined contrast threshold. In some embodiments, the rank is areal number ranging from 0 to N−1, N being a total number of subjects inthe plurality of subjects. In some embodiments, the rank is a realnumber ranging from 0 to N−1, N being a total number of retests of theplurality of subjects. In some embodiments, the distance is one or moreselected from: a Euclidean distance, a Manhattan distance, and aMahalanobis distance. In some embodiments, assessing the rank precisioncomprises: calculating a normalized rank, the normalized rank being therank of the distance divided by a total number of subjects of theplurality of subjects; and calculating the rank precision, the rankprecision being equal to one subtracted by the normalized rank. In someembodiments, the rank precision is an inverse of the rank of the retestresult of the first subject. In some embodiments, the test-retestprecision is mean of the rank precision for each of the plurality ofsubjects. In some embodiments, the test-retest precision is mean of therank precision for each of the plurality of retest. In some embodiments,the rank is a real number ranging from 1 to N, N being a total number ofsubjects of the plurality of subjects. In some embodiments, the rank isa real number ranging from 1 to N, N being a total number of retests ofthe plurality of subjects. In some embodiments, the vision test machineis one or more selected from: a computerized adaptive contrastsensitivity testing device, a quick CSF (qCSF) testing device, a OCTmachine, a MRI machine, an ultrasound machine, a visual field testingmachine, a fundus photography system, a dark adaptation measurementmachine, an auto-refractor machine, a frequency-doubling thresholdmachine, a tonometer machine, an aberrometer machine, an eye-trackingdevice, and an ocular alignment machine.

In another aspect, disclosed herein are computer-implemented systemscomprising a digital processing device comprising at least oneprocessor, an operating system configured to perform executableinstructions, a memory, and a computer program including instructionsexecutable by the digital processing device to create an application ofevaluating test-retest precision of a test and a retest of a pluralityof subjects using fractional rank precision (FRP) or mean-averageprecision (MAP), comprising: a) a software module configured to collecta test result and a retest result from each of the plurality ofsubjects, wherein the test result and the retest result are described inone or more feature spaces and one of the test result and the retestresult is collected from a vision test machine; b) a software moduleconfigured to select a first subject from the plurality of subjects anda first test result of the first subject; c) a software moduleconfigured to calculate distances from the first test result to theretest result of each of the plurality of subjects; d) a software moduleconfigured to assess a similarity between the first test result and theretest result of each of the plurality of subjects by ranking thedistances in a non-descending order; e) a software module configured toassess a rank precision for the first subject based on a rank of adistance from the first test result to the retest result of the firstsubject; f) a software module configured to repeat b), c), d), and e)for each of the plurality of subjects; and g) a software moduleconfigured to evaluate the test-retest precision based on the rankprecision for each of the plurality of subjects. In some embodiments,the test is a first vision test. In some embodiments, the retest is thefirst vision test or a second vision test. In some embodiments, thefirst vision test or the second vision test is one or more selectedfrom: a vision acuity test, a CSF test, and an OCT test. In someembodiments, the feature space is one-dimensional or multi-dimensional.In some embodiments, the feature space comprises a feature. In someembodiments, the feature includes one or more features selected from: amedian AULCSF computed over the spatial frequency range of 1.5 to 6 cpd,a median AULCSF computed over the spatial frequency range of 6 to 12cpd, a median AULCSF computed over the spatial frequency range of 12 to18 cpd, a median AULCSF computed over the spatial frequency range of 1.5to 18 cpd, a CSF acuity, a parameter of CSF, a contrast sensitivity forat least one spatial frequency selected from 1, 1.5, 3, 6, 12, and 18cpd, a peak sensitivity of the CSF, and a spatial frequency at which aCSF reaches a pre-determined contrast threshold. In some embodiments,the rank is a real number ranging from 0 to N−1, N being a total numberof subjects in the plurality of subjects. In some embodiments, the rankis a real number ranging from 0 to N−1, N being a total number ofretests of the plurality of subjects. In some embodiments, the distanceis one or more selected from: a Euclidean distance, a Manhattandistance, and a Mahalanobis distance. In some embodiments, assessing therank precision comprises: calculating a normalized rank, the normalizedrank being the rank of the distance divided by a total number ofsubjects of the plurality of subjects; and calculating the rankprecision, the rank precision being equal to one subtracted by thenormalized rank. In some embodiments, the rank precision is an inverseof the rank of the retest result of the first subject. In someembodiments, the test-retest precision is mean of the rank precision foreach of the plurality of subjects. In some embodiments, the test-retestprecision is mean of the rank precision for each of the plurality ofretest. In some embodiments, the rank is a real number ranging from 1 toN, N being a total number of subjects of the plurality of subjects. Insome embodiments, the rank is a real number ranging from 1 to N, N beinga total number of retests of the plurality of subjects. In someembodiments, the vision test machine is one or more selected from: acomputerized adaptive contrast sensitivity testing device, a qCSFtesting device, a OCT machine, a MRI machine, an ultrasound machine, avisual field testing machine, a fundus photography system, a darkadaptation measurement machine, an auto-refractor machine, afrequency-doubling threshold machine, a tonometer machine, anaberrometer machine, an eye-tracking device, and an ocular alignmentmachine.

In another aspect, disclosed herein are non-transitory computer-readablestorage media encoded with a computer program including instructionsexecutable by a processor to create an application for evaluatingtest-retest precision of a test and a retest of a plurality of subjectsusing fractional rank precision (FRP) or mean-average precision (MAP),comprising: a) a database, in a computer memory, of a test result and aretest result for each of a plurality of subjects, wherein the testresult and the retest result are described in one or more featurespaces, and one of the test result and the retest result is obtainedfrom a vision test machine; b) a software module configured to select afirst subject from the plurality of subjects and a first test result ofthe first subject; c) a software module configured to calculatedistances from the first test result to the retest result of each of theplurality of subjects; d) a software module configured to assess asimilarity between the first test result and the retest result of eachof the plurality of subjects by ranking the distances in anon-descending order; e) a software module configured to assess a rankprecision for the first subject based on a rank of a distance from thefirst test result to the retest result of the first subject; f) asoftware module configured to repeat b), c), d), and e) for each of theplurality of subjects; and g) a software module configured to evaluatethe test-retest precision based on the rank precision for each of theplurality of subjects. In some embodiments, the test is a first visiontest. In some embodiments, the retest is the first vision test or asecond vision test. In some embodiments, the first vision test or thesecond vision test is one or more selected from: a vision acuity test, aCSF test, and an OCT test. In some embodiments, the feature space isone-dimensional or multi-dimensional. In some embodiments, the featurespace comprises a feature. In some embodiments, the feature includes oneor more features selected from: a median AULCSF computed over thespatial frequency range of 1.5 to 6 cpd, a median AULCSF computed overthe spatial frequency range of 6 to 12 cpd, a median AULCSF computedover the spatial frequency range of 12 to 18 cpd, a median AULCSFcomputed over the spatial frequency range of 1.5 to 18 cpd, a CSFacuity, a parameter of CSF, a contrast sensitivity for at least onespatial frequency selected from 1, 1.5, 3, 6, 12, and 18 cpd, a peaksensitivity of the CSF, and a spatial frequency at which a CSF reaches apre-determined contrast threshold. In some embodiments, the rank is areal number ranging from 0 to N−1, N being a total number of subjects inthe plurality of subjects. In some embodiments, the rank is a realnumber ranging from 0 to N−1, N being a total number of retests of theplurality of subjects. In some embodiments, the distance is one or moreselected from: a Euclidean distance, a Manhattan distance, and aMahalanobis distance. In some embodiments, assessing the rank precisioncomprises: calculating a normalized rank, the normalized rank being therank of the distance divided by a total number of subjects of theplurality of subjects; and calculating the rank precision, the rankprecision being equal to one subtracted by the normalized rank. In someembodiments, the rank precision is an inverse of the rank of the retestresult of the first subject. In some embodiments, the test-retestprecision is mean of the rank precision for each of the plurality ofsubjects. In some embodiments, the test-retest precision is mean of therank precision for each of the plurality of retest. In some embodiments,the rank is a real number ranging from 1 to N, N being a total number ofsubjects of the plurality of subjects. In some embodiments, the rank isa real number ranging from 1 to N, N being a total number of retests ofthe plurality of subjects. In some embodiments, the vision test machineis one or more selected from: a computerized adaptive contrastsensitivity testing device, a qCSF testing device, a OCT machine, a MRImachine, an ultrasound machine, a visual field testing machine, a fundusphotography system, a dark adaptation measurement machine, anauto-refractor machine, a frequency-doubling threshold machine, atonometer machine, an aberrometer machine, an eye-tracking device, andan ocular alignment machine.

In another aspect, disclosed herein are computer-implemented methods ofevaluating test-retest precision of a test and a retest of a pluralityof subjects using fractional rank precision (FRP) or mean-averageprecision (MAP), comprising: a) collecting, by a computer, a test resultand a plurality of retest results from each of the plurality ofsubjects, wherein the test result and the plurality of retest resultsare described in one or more feature spaces, and one of the test resultand the plurality of retest results is collected from a vision testmachine; b) selecting, by the computer, a first subject from theplurality of subjects and a first test result of the first subject; c)calculating, by the computer, distances from the first test result toeach of the plurality of retest results of each of the plurality ofsubjects; d) assessing, by the computer, a similarity between the firsttest result and the plurality of retest results of each of the pluralityof subjects by ranking the distances in a non-descending order; e)assessing, by the computer, a rank precision for the first subject basedon the rank of distances from the first test result to each of theplurality of retest results of the first subject; f) repeating, by thecomputer, steps b), c), d), and e) for each of the plurality ofsubjects; and g) evaluating, by the computer, the test-retest precisionbased on the rank precision for each of the plurality of subjects. Insome embodiments, the test is a first vision test. In some embodiments,the retest is the first vision test or a second vision test. In someembodiments, the first vision test or the second vision test is one ormore selected from: a vision acuity test, a CSF test, and an OCT test.In some embodiments, the feature space is one-dimensional ormulti-dimensional. In some embodiments, the feature space comprises afeature. In some embodiments, the feature includes one or more featuresselected from: a median AULCSF computed over the spatial frequency rangeof 1.5 to 6 cpd, a median AULCSF computed over the spatial frequencyrange of 6 to 12 cpd, a median AULCSF computed over the spatialfrequency range of 12 to 18 cpd, a median AULCSF computed over thespatial frequency range of 1.5 to 18 cpd, a CSF acuity, a parameter ofCSF, a contrast sensitivity for at least one spatial frequency selectedfrom 1, 1.5, 3, 6, 12, and 18 cpd, a peak sensitivity of the CSF, and aspatial frequency at which a CSF reaches a pre-determined contrastthreshold. In some embodiments, the rank is a real number ranging from 0to N−1, N being a total number of subjects in the plurality of subjects.In some embodiments, the rank is a real number ranging from 0 to N−1, Nbeing a total number of retests of the plurality of subjects. In someembodiments, the distance is one or more selected from: a Euclideandistance, a Manhattan distance, and a Mahalanobis distance. In someembodiments, assessing the rank precision comprises: calculating anormalized rank, the normalized rank being the rank of the distancedivided by a total number of subjects of the plurality of subjects; andcalculating the rank precision, the rank precision being equal to onesubtracted by the normalized rank. In some embodiments, the rankprecision is an inverse of the rank of the retest result of the firstsubject. In some embodiments, the test-retest precision is mean of therank precision for each of the plurality of subjects. In someembodiments, the test-retest precision is mean of the rank precision foreach of the plurality of retest. In some embodiments, the rank is a realnumber ranging from 1 to N, N being a total number of subjects of theplurality of subjects. In some embodiments, the rank is a real numberranging from 1 to N, N being a total number of retests of the pluralityof subjects. In some embodiments, the vision test machine is one or moreselected from: a computerized adaptive contrast sensitivity testingdevice, a qCSF testing device, a OCT machine, a MRI machine, anultrasound machine, a visual field testing machine, a fundus photographysystem, a dark adaptation measurement machine, an auto-refractormachine, a frequency-doubling threshold machine, a tonometer machine, anaberrometer machine, an eye-tracking device, and an ocular alignmentmachine.

In another aspect, disclosed herein are computer-implemented systemscomprising a digital processing device comprising at least oneprocessor, an operating system configured to perform executableinstructions, a memory, and a computer program including instructionsexecutable by the digital processing device to create an application ofevaluating test-retest precision of a test and a retest of a pluralityof subjects using fractional rank precision (FRP) or mean-averageprecision (MAP), comprising: a) a software module configured to collecta test result and a plurality of retest results from each of theplurality of subjects, wherein the test result and the plurality ofretest results are described in one or more feature spaces, and one ofthe test result and the plurality of retest results is collected from avision test machine; b) a software module configured to select a firstsubject from the plurality of subjects and a first test result of thefirst subject; c) a software module configured to calculate distancesfrom the first test result to each of the plurality of retest results ofeach of the plurality of subjects; d) a software module configured toassess a similarity between the first test result and the plurality ofretest results of each of the plurality of subjects by ranking thedistances in a non-descending order; e) a software module configured toassess a rank precision for the first subject based on the rank ofdistances from the first test result to each of the plurality of retestresults of the first subject; f) a software module configured to repeatb), c), d), and e) for each of the plurality of subjects; and g) asoftware module configured to evaluate the test-retest precision basedon the rank precision for each of the plurality of subjects. In someembodiments, the test is a first vision test. In some embodiments, theretest is the first vision test or a second vision test. In someembodiments, the first vision test or the second vision test is one ormore selected from: a vision acuity test, a CSF test, and an OCT test.In some embodiments, the feature space is one-dimensional ormulti-dimensional. In some embodiments, the feature space comprises afeature. In some embodiments, the feature includes one or more featuresselected from: a median AULCSF computed over the spatial frequency rangeof 1.5 to 6 cpd, a median AULCSF computed over the spatial frequencyrange of 6 to 12 cpd, a median AULCSF computed over the spatialfrequency range of 12 to 18 cpd, a median AULCSF computed over thespatial frequency range of 1.5 to 18 cpd, a CSF acuity, a parameter ofCSF, a contrast sensitivity for at least one spatial frequency selectedfrom 1, 1.5, 3, 6, 12, and 18 cpd, a peak sensitivity of the CSF, and aspatial frequency at which a CSF reaches a pre-determined contrastthreshold. In some embodiments, the rank is a real number ranging from 0to N−1, N being a total number of subjects in the plurality of subjects.In some embodiments, the rank is a real number ranging from 0 to N−1, Nbeing a total number of retests of the plurality of subjects. In someembodiments, the distance is one or more selected from: a Euclideandistance, a Manhattan distance, and a Mahalanobis distance. In someembodiments, assessing the rank precision comprises: calculating anormalized rank, the normalized rank being the rank of the distancedivided by a total number of subjects of the plurality of subjects; andcalculating the rank precision, the rank precision being equal to onesubtracted by the normalized rank. In some embodiments, the rankprecision is an inverse of the rank of the retest result of the firstsubject. In some embodiments, the test-retest precision is mean of therank precision for each of the plurality of subjects. In someembodiments, the test-retest precision is mean of the rank precision foreach of the plurality of retest. In some embodiments, the rank is a realnumber ranging from 1 to N, N being a total number of subjects of theplurality of subjects. In some embodiments, the rank is a real numberranging from 1 to N, N being a total number of retests of the pluralityof subjects. In some embodiments, the vision test machine is one or moreselected from: a computerized adaptive contrast sensitivity testingdevice, a qCSF testing device, a OCT machine, a MRI machine, anultrasound machine, a visual field testing machine, a fundus photographysystem, a dark adaptation measurement machine, an auto-refractormachine, a frequency-doubling threshold machine, a tonometer machine, anaberrometer machine, an eye-tracking device, and an ocular alignmentmachine.

In yet another aspect, disclosed herein are non-transitorycomputer-readable storage media encoded with a computer programincluding instructions executable by a processor to create anapplication for evaluating test-retest precision of a test and a retestof a plurality of subjects using fractional rank precision (FRP) ormean-average precision (MAP), comprising: a) a database, in a computermemory, of a test result and a plurality of retest results from each ofthe plurality of subjects, wherein the test result and the plurality ofretest results are described in one or more feature spaces, and one ofthe test result and the plurality of retest results is collected from avision test machine; b) a software module configured to select a firstsubject from the plurality of subjects and a first test result of thefirst subject; c) a software module configured to calculate distancesfrom the first test result to each of the plurality of retest results ofeach of the plurality of subjects; d) a software module configured toassess a similarity between the first test result and the plurality ofretest results of each of the plurality of subjects by ranking thedistances in a non-descending order; e) a software module configured toassess a rank precision for the first subject based on the rank ofdistances from the first test result to each of the plurality of retestresults of the first subject; f) a software module configured to repeatb), c), d), and e) for each of the plurality of subjects; and g) asoftware module configured to evaluate the test-retest precision basedon the rank precision for each of the plurality of subjects. In someembodiments, the test is a first vision test. In some embodiments, theretest is the first vision test or a second vision test. In someembodiments, the first vision test or the second vision test is one ormore selected from: a vision acuity test, a CSF test, and an OCT test.In some embodiments, the feature space is one-dimensional ormulti-dimensional. In some embodiments, the feature space comprises afeature. In some embodiments, the feature includes one or more featuresselected from: a median AULCSF computed over the spatial frequency rangeof 1.5 to 6 cpd, a median AULCSF computed over the spatial frequencyrange of 6 to 12 cpd, a median AULCSF computed over the spatialfrequency range of 12 to 18 cpd, a median AULCSF computed over thespatial frequency range of 1.5 to 18 cpd, a CSF acuity, a parameter ofCSF, a contrast sensitivity for at least one spatial frequency selectedfrom 1, 1.5, 3, 6, 12, and 18 cpd, a peak sensitivity of the CSF, and aspatial frequency at which a CSF reaches a pre-determined contrastthreshold. In some embodiments, the rank is a real number ranging from 0to N−1, N being a total number of subjects in the plurality of subjects.In some embodiments, the rank is a real number ranging from 0 to N−1, Nbeing a total number of retests of the plurality of subjects. In someembodiments, the distance is one or more selected from: a Euclideandistance, a Manhattan distance, and a Mahalanobis distance. In someembodiments, assessing the rank precision comprises: calculating anormalized rank, the normalized rank being the rank of the distancedivided by a total number of subjects of the plurality of subjects; andcalculating the rank precision, the rank precision being equal to onesubtracted by the normalized rank. In some embodiments, the rankprecision is an inverse of the rank of the retest result of the firstsubject. In some embodiments, the test-retest precision is mean of therank precision for each of the plurality of subjects. In someembodiments, the test-retest precision is mean of the rank precision foreach of the plurality of retest. In some embodiments, the rank is a realnumber ranging from 1 to N, N being a total number of subjects of theplurality of subjects. In some embodiments, the rank is a real numberranging from 1 to N, N being a total number of retests of the pluralityof subjects. In some embodiments, the vision test machine is one or moreselected from: a computerized adaptive contrast sensitivity testingdevice, a qCSF testing device, a OCT machine, a MRI machine, anultrasound machine, a visual field testing machine, a fundus photographysystem, a dark adaptation measurement machine, an auto-refractormachine, a frequency-doubling threshold machine, a tonometer machine, anaberrometer machine, an eye-tracking device, and an ocular alignmentmachine.

In yet another aspect, disclose herein are computer-implemented methodsof evaluating test-retest precision of a test and a retest of aplurality of subjects using fractional rank precision (FRP) ormean-average precision (MAP), comprising: a) collecting, by a computer,a test result and a retest result of each of the plurality of subjects,wherein the test result and the retest result are described in one ormore feature spaces and one of the test result and the retest result iscollected from a vision test machine; b) selecting, by the computer, anth subject from the plurality of subjects and the test result of thenth subject; c) calculating, by the computer, distances from the testresult of the nth subject to the retest result of each of the pluralityof subjects; d) assessing, by the computer, a similarity between thetest result of the nth subject and the retest result of each of theplurality of subjects by ranking the distances in a non-descendingorder; e) assessing, by the computer, a rank precision for the nthsubject based on the rank of the distance from the test result of thenth subject to the retest result of the nth subject; f) repeating, bythe computer, steps b), c), d), and e) for 1≤n≤N, N being a total numberof subjects in the plurality of subjects; and g) evaluating, by thecomputer, the test-retest precision based on the rank precision for eachof the plurality of subjects. In some embodiments, the test is a firstvision test. In some embodiments, the retest is the first vision test ora second vision test. In some embodiments, the first vision test or thesecond vision test is one or more selected from: a vision acuity test, aCSF test, and an OCT test. In some embodiments, the feature space isone-dimensional or multi-dimensional. In some embodiments, the featurespace comprises a feature. In some embodiments, the feature includes oneor more features selected from: a median AULCSF computed over thespatial frequency range of 1.5 to 6 cpd, a median AULCSF computed overthe spatial frequency range of 6 to 12 cpd, a median AULCSF computedover the spatial frequency range of 12 to 18 cpd, a median AULCSFcomputed over the spatial frequency range of 1.5 to 18 cpd, a CSFacuity, a parameter of CSF, a contrast sensitivity for at least onespatial frequency selected from 1, 1.5, 3, 6, 12, and 18 cpd, a peaksensitivity of the CSF, and a spatial frequency at which a CSF reaches apre-determined contrast threshold. In some embodiments, the rank is areal number ranging from 0 to N−1, N being a total number of subjects inthe plurality of subjects. In some embodiments, the rank is a realnumber ranging from 0 to N−1, N being a total number of retests of theplurality of subjects. In some embodiments, the distance is one or moreselected from: a Euclidean distance, a Manhattan distance, and aMahalanobis distance. In some embodiments, assessing the rank precisioncomprises: calculating a normalized rank, the normalized rank being therank of the distance divided by a total number of subjects of theplurality of subjects; and calculating the rank precision, the rankprecision being equal to one subtracted by the normalized rank. In someembodiments, the rank precision is an inverse of the rank of the retestresult of the first subject. In some embodiments, the test-retestprecision is mean of the rank precision for each of the plurality ofsubjects. In some embodiments, the test-retest precision is mean of therank precision for each of the plurality of retest. In some embodiments,the rank is a real number ranging from 1 to N, N being a total number ofsubjects of the plurality of subjects. In some embodiments, the rank isa real number ranging from 1 to N, N being a total number of retests ofthe plurality of subjects. In some embodiments, the vision test machineis one or more selected from: a computerized adaptive contrastsensitivity testing device, a qCSF testing device, a OCT machine, a MRImachine, an ultrasound machine, a visual field testing machine, a fundusphotography system, a dark adaptation measurement machine, anauto-refractor machine, a frequency-doubling threshold machine, atonometer machine, an aberrometer machine, an eye-tracking device, andan ocular alignment machine.

In yet another aspect, disclosed herein are computer-implemented systemscomprising a digital processing device comprising at least oneprocessor, an operating system configured to perform executableinstructions, a memory, and a computer program including instructionsexecutable by the digital processing device to create an application ofevaluating test-retest precision of a test and a retest of a pluralityof subjects using fractional rank precision (FRP) or mean-averageprecision (MAP), comprising: a) a software module configured to collecta test result and a retest result from each of the plurality ofsubjects, wherein the test result and the retest result are described inone or more feature spaces and one of the test result and the retestresult is collected from a vision test machine; b) a software moduleconfigured to select a nth subject from the plurality of subjects and atest result of the nth subject; c) a software module configured tocalculate distances from the test result of the nth subject to theretest result of each of the plurality of subjects; d) a software moduleconfigured to assess a similarity between the test result of the nthsubject and the retest result of each of the plurality of subjects byranking the distances in a non-descending order; e) a software moduleconfigured to assess a rank precision for the nth subject based on arank of a distance from the test result of the nth subject to the retestresult of the nth subject; f) repeating, by the computer, steps b), c),d), and e) for 1≤n≤N, N being a total number of subjects in theplurality of subjects; and g) a software module configured to evaluatethe test-retest precision based on the rank precision for each of theplurality of subjects. In some embodiments, the test is a first visiontest. In some embodiments, the retest is the first vision test or asecond vision test. In some embodiments, the first vision test or thesecond vision test is one or more selected from: a vision acuity test, aCSF test, and an OCT test. In some embodiments, the feature space isone-dimensional or multi-dimensional. In some embodiments, the featurespace comprises a feature. In some embodiments, the feature includes oneor more features selected from: a median AULCSF computed over thespatial frequency range of 1.5 to 6 cpd, a median AULCSF computed overthe spatial frequency range of 6 to 12 cpd, a median AULCSF computedover the spatial frequency range of 12 to 18 cpd, a median AULCSFcomputed over the spatial frequency range of 1.5 to 18 cpd, a CSFacuity, a parameter of CSF, a contrast sensitivity for at least onespatial frequency selected from 1, 1.5, 3, 6, 12, and 18 cpd, a peaksensitivity of the CSF, and a spatial frequency at which a CSF reaches apre-determined contrast threshold. In some embodiments, the rank is areal number ranging from 0 to N−1, N being a total number of subjects inthe plurality of subjects. In some embodiments, the rank is a realnumber ranging from 0 to N−1, N being a total number of retests of theplurality of subjects. In some embodiments, the distance is one or moreselected from: a Euclidean distance, a Manhattan distance, and aMahalanobis distance. In some embodiments, assessing the rank precisioncomprises: calculating a normalized rank, the normalized rank being therank of the distance divided by a total number of subjects of theplurality of subjects; and calculating the rank precision, the rankprecision being equal to one subtracted by the normalized rank. In someembodiments, the rank precision is an inverse of the rank of the retestresult of the first subject. In some embodiments, the test-retestprecision is mean of the rank precision for each of the plurality ofsubjects. In some embodiments, the test-retest precision is mean of therank precision for each of the plurality of retest. In some embodiments,the rank is a real number ranging from 1 to N, N being a total number ofsubjects of the plurality of subjects. In some embodiments, the rank isa real number ranging from 1 to N, N being a total number of retests ofthe plurality of subjects. In some embodiments, the vision test machineis one or more selected from: a computerized adaptive contrastsensitivity testing device, a qCSF testing device, a OCT machine, a MRImachine, an ultrasound machine, a visual field testing machine, a fundusphotography system, a dark adaptation measurement machine, anauto-refractor machine, a frequency-doubling threshold machine, atonometer machine, an aberrometer machine, an eye-tracking device, andan ocular alignment machine.

In yet another aspect, disclosed herein are computer-implemented methodsof evaluating test-retest precision of a test and a retest of aplurality of subjects using fractional rank precision (FRP) ormean-average precision (MAP), comprising: a) collecting, by a computer,a test result and a plurality of retest results from each of theplurality of subjects, wherein the test result and the plurality ofretest results are described in one or more feature spaces, and one ofthe test result and the plurality of retest results is collected from avision test machine; b) selecting, by the computer, a nth subject fromthe plurality of subjects and a test result of the nth subject; c)calculating, by the computer, distances from the test result of the nthsubject to each of the plurality of retest results of each of theplurality of subjects; d) assessing, by the computer, a similaritybetween the test result of the nth subject and the plurality of retestresults of each of the plurality of subjects by ranking the distances ina non-descending order; e) assessing, by the computer, a rank precisionfor the first subject based on the rank of distances from the testresult of the nth subject to each of the plurality of retest results ofthe first subject; f) repeating, by the computer, steps b), c), d), ande) for 1≤n≤N, N being a total number of subjects in the plurality ofsubjects; and g) evaluating, by the computer, the test-retest precisionbased on the rank precision for each of the plurality of subjects. Insome embodiments, the test is a first vision test. In some embodiments,the retest is the first vision test or a second vision test. In someembodiments, the first vision test or the second vision test is one ormore selected from: a vision acuity test, a CSF test, and an OCT test.In some embodiments, the feature space is one-dimensional ormulti-dimensional. In some embodiments, the feature space comprises afeature. In some embodiments, the feature includes one or more featuresselected from: a median AULCSF computed over the spatial frequency rangeof 1.5 to 6 cpd, a median AULCSF computed over the spatial frequencyrange of 6 to 12 cpd, a median AULCSF computed over the spatialfrequency range of 12 to 18 cpd, a median AULCSF computed over thespatial frequency range of 1.5 to 18 cpd, a CSF acuity, a parameter ofCSF, a contrast sensitivity for at least one spatial frequency selectedfrom 1, 1.5, 3, 6, 12, and 18 cpd, a peak sensitivity of the CSF, and aspatial frequency at which a CSF reaches a pre-determined contrastthreshold. In some embodiments, the rank is a real number ranging from 0to N−1, N being a total number of subjects in the plurality of subjects.In some embodiments, the rank is a real number ranging from 0 to N−1, Nbeing a total number of retests of the plurality of subjects. In someembodiments, the distance is one or more selected from: a Euclideandistance, a Manhattan distance, and a Mahalanobis distance. In someembodiments, assessing the rank precision comprises: calculating anormalized rank, the normalized rank being the rank of the distancedivided by a total number of subjects of the plurality of subjects; andcalculating the rank precision, the rank precision being equal to onesubtracted by the normalized rank. In some embodiments, the rankprecision is an inverse of the rank of the retest result of the firstsubject. In some embodiments, the test-retest precision is mean of therank precision for each of the plurality of subjects. In someembodiments, the test-retest precision is mean of the rank precision foreach of the plurality of retest. In some embodiments, the rank is a realnumber ranging from 1 to N, N being a total number of subjects of theplurality of subjects. In some embodiments, the rank is a real numberranging from 1 to N, N being a total number of retests of the pluralityof subjects. In some embodiments, the vision test machine is one or moreselected from: a computerized adaptive contrast sensitivity testingdevice, a qCSF testing device, a OCT machine, a MRI machine, anultrasound machine, a visual field testing machine, a fundus photographysystem, a dark adaptation measurement machine, an auto-refractormachine, a frequency-doubling threshold machine, a tonometer machine, anaberrometer machine, an eye-tracking device, and an ocular alignmentmachine.

In yet another aspect disclosed herein are computer-implemented systemscomprising a digital processing device comprising at least oneprocessor, an operating system configured to perform executableinstructions, a memory, and a computer program including instructionsexecutable by the digital processing device to create an application ofevaluating test-retest precision of a test and a retest of a pluralityof subjects using fractional rank precision (FRP) or mean-averageprecision (MAP), comprising: a) a software module configured to collecta test result and a retest result from each of the plurality ofsubjects, wherein the test result and the retest result are described inone or more feature spaces and one of the test result and the retestresult is collected from a vision test machine; b) a software moduleconfigured to select a nth subject from the plurality of subjects and atest result of the nth subject; c) a software module configured tocalculate distances from the test result of the nth subject to theretest result of each of the plurality of subjects; d) a software moduleconfigured to assess a similarity between the test result of the nthsubject and the retest result of each of the plurality of subjects byranking the distances in a non-descending order; e) a software moduleconfigured to assess a rank precision for the nth subject based on arank of a distance from the test result of the nth subject to the retestresult of the nth subject; f) repeating, by the computer, steps b), c),d), and e) for 1≤n≤N, N being a total number of subjects in theplurality of subjects; and g) a software module configured to evaluatethe test-retest precision based on the rank precision for each of theplurality of subjects. In some embodiments, the test is a first visiontest. In some embodiments, the retest is the first vision test or asecond vision test. In some embodiments, the first vision test or thesecond vision test is one or more selected from: a vision acuity test, aCSF test, and an OCT test. In some embodiments, the feature space isone-dimensional or multi-dimensional. In some embodiments, the featurespace comprises a feature. In some embodiments, the feature includes oneor more features selected from: a median AULCSF computed over thespatial frequency range of 1.5 to 6 cpd, a median AULCSF computed overthe spatial frequency range of 6 to 12 cpd, a median AULCSF computedover the spatial frequency range of 12 to 18 cpd, a median AULCSFcomputed over the spatial frequency range of 1.5 to 18 cpd, a CSFacuity, a parameter of CSF, a contrast sensitivity for at least onespatial frequency selected from 1, 1.5, 3, 6, 12, and 18 cpd, a peaksensitivity of the CSF, and a spatial frequency at which a CSF reaches apre-determined contrast threshold. In some embodiments, the rank is areal number ranging from 0 to N−1, N being a total number of subjects inthe plurality of subjects. In some embodiments, the rank is a realnumber ranging from 0 to N−1, N being a total number of retests of theplurality of subjects. In some embodiments, the distance is one or moreselected from: a Euclidean distance, a Manhattan distance, and aMahalanobis distance. In some embodiments, assessing the rank precisioncomprises: calculating a normalized rank, the normalized rank being therank of the distance divided by a total number of subjects of theplurality of subjects; and calculating the rank precision, the rankprecision being equal to one subtracted by the normalized rank. In someembodiments, the rank precision is an inverse of the rank of the retestresult of the first subject. In some embodiments, the test-retestprecision is mean of the rank precision for each of the plurality ofsubjects. In some embodiments, the test-retest precision is mean of therank precision for each of the plurality of retest. In some embodiments,the rank is a real number ranging from 1 to N, N being a total number ofsubjects of the plurality of subjects. In some embodiments, the rank isa real number ranging from 1 to N, N being a total number of retests ofthe plurality of subjects. In some embodiments, the vision test machineis one or more selected from: a computerized adaptive contrastsensitivity testing device, a qCSF testing device, a OCT machine, a MRImachine, an ultrasound machine, a visual field testing machine, a fundusphotography system, a dark adaptation measurement machine, anauto-refractor machine, a frequency-doubling threshold machine, atonometer machine, an aberrometer machine, an eye-tracking device, andan ocular alignment machine.

In yet another aspect, disclosed herein are computer-implemented systemscomprising a digital processing device comprising at least oneprocessor, an operating system configured to perform executableinstructions, a memory, and a computer program including instructionsexecutable by the digital processing device to create an application ofevaluating test-retest precision of a test and a retest of a pluralityof subjects using fractional rank precision (FRP) or mean-averageprecision (MAP), comprising: a) a software module configured to collecta test result and a plurality of retest results from each of theplurality of subjects, wherein the test result and the plurality ofretest results are described in one or more feature spaces, and one ofthe test result and the plurality of retest results is collected from avision test machine; b) a software module configured to select a nthsubject from the plurality of subjects and a test result of the nthsubject; c) a software module configured to calculate distances from thetest result of the nth subject to each of the plurality of retestresults of each of the plurality of subjects; d) a software moduleconfigured to assess a similarity between the test result of the nthsubject and the plurality of retest results of each of the plurality ofsubjects by ranking the distances in a non-descending order; e) asoftware module configured to assess a rank precision for the nthsubject based on the rank of distances from the test result of the nthsubject to each of the plurality of retest results of the nth subject;f) a software module configured to repeat b), c), d), and e) for 1≤n≤N,N being a total number of subjects in the plurality of subjects; and g)a software module configured to evaluate the test-retest precision basedon the rank precision for each of the plurality of subjects. In someembodiments, the test is a first vision test. In some embodiments, theretest is the first vision test or a second vision test. In someembodiments, the first vision test or the second vision test is one ormore selected from: a vision acuity test, a CSF test, and an OCT test.In some embodiments, the feature space is one-dimensional ormulti-dimensional. In some embodiments, the feature space comprises afeature. In some embodiments, the feature includes one or more featuresselected from: a median AULCSF computed over the spatial frequency rangeof 1.5 to 6 cpd, a median AULCSF computed over the spatial frequencyrange of 6 to 12 cpd, a median AULCSF computed over the spatialfrequency range of 12 to 18 cpd, a median AULCSF computed over thespatial frequency range of 1.5 to 18 cpd, a CSF acuity, a parameter ofCSF, a contrast sensitivity for at least one spatial frequency selectedfrom 1, 1.5, 3, 6, 12, and 18 cpd, a peak sensitivity of the CSF, and aspatial frequency at which a CSF reaches a pre-determined contrastthreshold. In some embodiments, the rank is a real number ranging from 0to N−1, N being a total number of subjects in the plurality of subjects.In some embodiments, the rank is a real number ranging from 0 to N−1, Nbeing a total number of retests of the plurality of subjects. In someembodiments, the distance is one or more selected from: a Euclideandistance, a Manhattan distance, and a Mahalanobis distance. In someembodiments, assessing the rank precision comprises: calculating anormalized rank, the normalized rank being the rank of the distancedivided by a total number of subjects of the plurality of subjects; andcalculating the rank precision, the rank precision being equal to onesubtracted by the normalized rank. In some embodiments, the rankprecision is an inverse of the rank of the retest result of the firstsubject. In some embodiments, the test-retest precision is mean of therank precision for each of the plurality of subjects. In someembodiments, the test-retest precision is mean of the rank precision foreach of the plurality of retest. In some embodiments, the rank is a realnumber ranging from 1 to N, N being a total number of subjects of theplurality of subjects. In some embodiments, the rank is a real numberranging from 1 to N, N being a total number of retests of the pluralityof subjects. In some embodiments, the vision test machine is one or moreselected from: a computerized adaptive contrast sensitivity testingdevice, a qCSF testing device, a OCT machine, a MRI machine, anultrasound machine, a visual field testing machine, a fundus photographysystem, a dark adaptation measurement machine, an auto-refractormachine, a frequency-doubling threshold machine, a tonometer machine, anaberrometer machine, an eye-tracking device, and an ocular alignmentmachine.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows a non-limiting example of the Mean Average Precision methoddisclosed herein; in this case, obtaining a distance and a rank ofdistance based on test-retest results of a subject.

FIG. 2 shows a non-limiting example of effect of quantization of areaunder log of contrast sensitivity function (AULCSF) score on test-retestreliability measured by CoR, ICC, FRP, and MAP.

FIG. 3 shows a non-limiting example of the fractional rank precisionmethod disclosed herein; in this case, the test-retest precisiondistributions across subjects for different vision sensitivity features.

FIG. 4 shows a non-limiting example of effect of quantization of areaunder log of contrast sensitivity function (AULCSF) score on test-retestreliability measured by CoR, ICC, FRP, and MAP.

FIG. 5 shows non-limiting examples of evaluation for standardtest-retest reliability methods, CoR and ICC.

FIG. 6 shows a non-limiting example of the fractional rank precisionmethod disclosed herein; in this case, fractional rank precision overthe time course for different vision-based features.

FIG. 7 shows a non-limiting example of the fractional rank precisionmethod disclosed herein; in this case, fractional rank precision overthe time course for different priors in quick CSF(qCSF) analysisinitialization.

FIG. 8 shows a non-limiting exemplary embodiment of a digital processingdevice as disclosed herein.

DETAILED DESCRIPTION OF THE INVENTION

The method and systems disclosed herein, in various embodiments, adaptat least one method in machine learning and/or information retrievalfield and applies it in the measurement of similarity in test-retestpairs of at least one subject. In some embodiments, the methods andsystems disclosed herein include at least a mean average precisionmethod (MAP) adapted from machine learning and/or information retrieval.In some embodiments, the methods and systems disclosed herein include atleast a fractional ranking precision (FRP) method derived from machinelearning and/or information retrieval field. The methods and systemsdisclosed herein, for non-limiting examples, MAP and FRP, improve thelimitations of traditional methods for the measurement of test-retestreliability and precision. Additionally, the methods and systemsdisclosed herein provide sensitivity (detection of subtle changes) androbustness (in the presence of artifacts) in evaluating theeffectiveness of vision-based features in the detection of criticalvision changes caused by disease progression and/or therapeuticinterventions. Furthermore, the methods and systems disclosed hereinenable assessment of vision-based features in multi-dimensional featurespace.

In one aspect, disclosed herein are computer-implemented methods ofevaluating test-retest precision of a test and a retest of a pluralityof subjects using fractional rank precision (FRP) or mean-averageprecision (MAP), comprising: a) collecting, by a computer, a test resultand a retest result of each of the plurality of subjects, wherein thetest result and the retest result are described in one or more featurespaces and one of the test result and the retest result is collectedfrom a vision test machine; b) selecting, by the computer, a firstsubject from the plurality of subjects and a first test result of thefirst subject; c) calculating, by the computer, distances from the firsttest result to the retest result of each of the plurality of subjects;d) assessing, by the computer, a similarity between the first testresult and the retest result of each of the plurality of subjects byranking the distances in a non-descending order; e) assessing, by thecomputer, a rank precision for the first subject based on a rank of adistance from the first test result to the retest result of the firstsubject; f) repeating, by the computer, steps b), c), d), and e) foreach of the plurality of subjects; and g) evaluating, by the computer,the test-retest precision based on the rank precision for each of theplurality of subjects. In some embodiments, the test is a first visiontest. In some embodiments, the retest is the first vision test or asecond vision test. In some embodiments, the first vision test or thesecond vision test is one or more selected from: a vision acuity test, aCSF test, and an OCT test. In some embodiments, the feature space isone-dimensional or multi-dimensional. In some embodiments, the featurespace comprises a feature. In some embodiments, the feature includes oneor more features selected from: a median AULCSF computed over thespatial frequency range of 1.5 to 6 cpd, a median AULCSF computed overthe spatial frequency range of 6 to 12 cpd, a median AULCSF computedover the spatial frequency range of 12 to 18 cpd, a median AULCSFcomputed over the spatial frequency range of 1.5 to 18 cpd, a CSFacuity, a parameter of CSF, a contrast sensitivity for at least onespatial frequency selected from 1, 1.5, 3, 6, 12, and 18 cpd, a peaksensitivity of the CSF, and a spatial frequency at which a CSF reaches apre-determined contrast threshold. In some embodiments, the rank is areal number ranging from 0 to N−1, N being a total number of subjects inthe plurality of subjects. In some embodiments, the rank is a realnumber ranging from 0 to N−1, N being a total number of retests of theplurality of subjects. In some embodiments, the distance is one or moreselected from: a Euclidean distance, a Manhattan distance, and aMahalanobis distance. In some embodiments, assessing the rank precisioncomprises: calculating a normalized rank, the normalized rank being therank of the distance divided by a total number of subjects of theplurality of subjects; and calculating the rank precision, the rankprecision being equal to one subtracted by the normalized rank. In someembodiments, the rank precision is an inverse of the rank of the retestresult of the first subject. In some embodiments, the test-retestprecision is mean of the rank precision for each of the plurality ofsubjects. In some embodiments, the test-retest precision is mean of therank precision for each of the plurality of retest. In some embodiments,the rank is a real number ranging from 1 to N, N being a total number ofsubjects of the plurality of subjects. In some embodiments, the rank isa real number ranging from 1 to N, N being a total number of retests ofthe plurality of subjects. In some embodiments, the vision test machineis one or more selected from: a computerized adaptive contrastsensitivity testing device, a qCSF testing device, a OCT machine, a MRImachine, an ultrasound machine, a visual field testing machine, a fundusphotography system, a dark adaptation measurement machine, anauto-refractor machine, a frequency-doubling threshold machine, atonometer machine, an aberrometer machine, an eye-tracking device, andan ocular alignment machine.

In another aspect, disclosed herein are computer-implemented systemscomprising a digital processing device comprising at least oneprocessor, an operating system configured to perform executableinstructions, a memory, and a computer program including instructionsexecutable by the digital processing device to create an application ofevaluating test-retest precision of a test and a retest of a pluralityof subjects using fractional rank precision (FRP) or mean-averageprecision (MAP), comprising: a) a software module configured to collecta test result and a retest result from each of the plurality ofsubjects, wherein the test result and the retest result are described inone or more feature spaces and one of the test result and the retestresult is collected from a vision test machine; b) a software moduleconfigured to select a first subject from the plurality of subjects anda first test result of the first subject; c) a software moduleconfigured to calculate distances from the first test result to theretest result of each of the plurality of subjects; d) a software moduleconfigured to assess a similarity between the first test result and theretest result of each of the plurality of subjects by ranking thedistances in a non-descending order; e) a software module configured toassess a rank precision for the first subject based on a rank of adistance from the first test result to the retest result of the firstsubject; f) a software module configured to repeat b), c), d), and e)for each of the plurality of subjects; and g) a software moduleconfigured to evaluate the test-retest precision based on the rankprecision for each of the plurality of subjects. In some embodiments,the test is a first vision test. In some embodiments, the retest is thefirst vision test or a second vision test. In some embodiments, thefirst vision test or the second vision test is one or more selectedfrom: a vision acuity test, a CSF test, and an OCT test. In someembodiments, the feature space is one-dimensional or multi-dimensional.In some embodiments, the feature space comprises a feature. In someembodiments, the feature includes one or more features selected from: amedian AULCSF computed over the spatial frequency range of 1.5 to 6 cpd,a median AULCSF computed over the spatial frequency range of 6 to 12cpd, a median AULCSF computed over the spatial frequency range of 12 to18 cpd, a median AULCSF computed over the spatial frequency range of 1.5to 18 cpd, a CSF acuity, a parameter of CSF, a contrast sensitivity forat least one spatial frequency selected from 1, 1.5, 3, 6, 12, and 18cpd, a peak sensitivity of the CSF, and a spatial frequency at which aCSF reaches a pre-determined contrast threshold. In some embodiments,the rank is a real number ranging from 0 to N−1, N being a total numberof subjects in the plurality of subjects. In some embodiments, the rankis a real number ranging from 0 to N−1, N being a total number ofretests of the plurality of subjects. In some embodiments, the distanceis one or more selected from: a Euclidean distance, a Manhattandistance, and a Mahalanobis distance. In some embodiments, assessing therank precision comprises: calculating a normalized rank, the normalizedrank being the rank of the distance divided by a total number ofsubjects of the plurality of subjects; and calculating the rankprecision, the rank precision being equal to one subtracted by thenormalized rank. In some embodiments, the rank precision is an inverseof the rank of the retest result of the first subject. In someembodiments, the test-retest precision is mean of the rank precision foreach of the plurality of subjects. In some embodiments, the test-retestprecision is mean of the rank precision for each of the plurality ofretest. In some embodiments, the rank is a real number ranging from 1 toN, N being a total number of subjects of the plurality of subjects. Insome embodiments, the rank is a real number ranging from 1 to N, N beinga total number of retests of the plurality of subjects. In someembodiments, the vision test machine is one or more selected from: acomputerized adaptive contrast sensitivity testing device, a qCSFtesting device, a OCT machine, a MRI machine, an ultrasound machine, avisual field testing machine, a fundus photography system, a darkadaptation measurement machine, an auto-refractor machine, afrequency-doubling threshold machine, a tonometer machine, anaberrometer machine, an eye-tracking device, and an ocular alignmentmachine.

In another aspect, disclosed herein are non-transitory computer-readablestorage media encoded with a computer program including instructionsexecutable by a processor to create an application for evaluatingtest-retest precision of a test and a retest of a plurality of subjectsusing fractional rank precision (FRP) or mean-average precision (MAP),comprising: a) a database, in a computer memory, of a test result and aretest result for each of a plurality of subjects, wherein the testresult and the retest result are described in one or more featurespaces, and one of the test result and the retest result is obtainedfrom a vision test machine; b) a software module configured to select afirst subject from the plurality of subjects and a first test result ofthe first subject; c) a software module configured to calculatedistances from the first test result to the retest result of each of theplurality of subjects; d) a software module configured to assess asimilarity between the first test result and the retest result of eachof the plurality of subjects by ranking the distances in anon-descending order; e) a software module configured to assess a rankprecision for the first subject based on a rank of a distance from thefirst test result to the retest result of the first subject; f) asoftware module configured to repeat b), c), d), and e) for each of theplurality of subjects; and g) a software module configured to evaluatethe test-retest precision based on the rank precision for each of theplurality of subjects. In some embodiments, the test is a first visiontest. In some embodiments, the retest is the first vision test or asecond vision test. In some embodiments, the first vision test or thesecond vision test is one or more selected from: a vision acuity test, aCSF test, and an OCT test. In some embodiments, the feature space isone-dimensional or multi-dimensional. In some embodiments, the featurespace comprises a feature. In some embodiments, the feature includes oneor more features selected from: a median AULCSF computed over thespatial frequency range of 1.5 to 6 cpd, a median AULCSF computed overthe spatial frequency range of 6 to 12 cpd, a median AULCSF computedover the spatial frequency range of 12 to 18 cpd, a median AULCSFcomputed over the spatial frequency range of 1.5 to 18 cpd, a CSFacuity, a parameter of CSF, a contrast sensitivity for at least onespatial frequency selected from 1, 1.5, 3, 6, 12, and 18 cpd, a peaksensitivity of the CSF, and a spatial frequency at which a CSF reaches apre-determined contrast threshold. In some embodiments, the rank is areal number ranging from 0 to N−1, N being a total number of subjects inthe plurality of subjects. In some embodiments, the rank is a realnumber ranging from 0 to N−1, N being a total number of retests of theplurality of subjects. In some embodiments, the distance is one or moreselected from: a Euclidean distance, a Manhattan distance, and aMahalanobis distance. In some embodiments, assessing the rank precisioncomprises: calculating a normalized rank, the normalized rank being therank of the distance divided by a total number of subjects of theplurality of subjects; and calculating the rank precision, the rankprecision being equal to one subtracted by the normalized rank. In someembodiments, the rank precision is an inverse of the rank of the retestresult of the first subject. In some embodiments, the test-retestprecision is mean of the rank precision for each of the plurality ofsubjects. In some embodiments, the test-retest precision is mean of therank precision for each of the plurality of retest. In some embodiments,the rank is a real number ranging from 1 to N, N being a total number ofsubjects of the plurality of subjects. In some embodiments, the rank isa real number ranging from 1 to N, N being a total number of retests ofthe plurality of subjects. In some embodiments, the vision test machineis one or more selected from: a computerized adaptive contrastsensitivity testing device, a qCSF testing device, a OCT machine, a MRImachine, an ultrasound machine, a visual field testing machine, a fundusphotography system, a dark adaptation measurement machine, anauto-refractor machine, a frequency-doubling threshold machine, atonometer machine, an aberrometer machine, an eye-tracking device, andan ocular alignment machine.

In another aspect, disclosed herein are computer-implemented methods ofevaluating test-retest precision of a test and a retest of a pluralityof subjects using fractional rank precision (FRP) or mean-averageprecision (MAP), comprising: a) collecting, by a computer, a test resultand a plurality of retest results from each of the plurality ofsubjects, wherein the test result and the plurality of retest resultsare described in one or more feature spaces, and one of the test resultand the plurality of retest results is collected from a vision testmachine; b) selecting, by the computer, a first subject from theplurality of subjects and a first test result of the first subject; c)calculating, by the computer, distances from the first test result toeach of the plurality of retest results of each of the plurality ofsubjects; d) assessing, by the computer, a similarity between the firsttest result and the plurality of retest results of each of the pluralityof subjects by ranking the distances in a non-descending order; e)assessing, by the computer, a rank precision for the first subject basedon the rank of distances from the first test result to each of theplurality of retest results of the first subject; f) repeating, by thecomputer, steps b), c), d), and e) for each of the plurality ofsubjects; and g) evaluating, by the computer, the test-retest precisionbased on the rank precision for each of the plurality of subjects.

In another aspect, disclosed herein are computer-implemented systemscomprising a digital processing device comprising at least oneprocessor, an operating system configured to perform executableinstructions, a memory, and a computer program including instructionsexecutable by the digital processing device to create an application ofevaluating test-retest precision of a test and a retest of a pluralityof subjects using fractional rank precision (FRP) or mean-averageprecision (MAP), comprising: a) a software module configured to collecta test result and a plurality of retest results from each of theplurality of subjects, wherein the test result and the plurality ofretest results are described in one or more feature spaces, and one ofthe test result and the plurality of retest results is collected from avision test machine; b) a software module configured to select a firstsubject from the plurality of subjects and a first test result of thefirst subject; c) a software module configured to calculate distancesfrom the first test result to each of the plurality of retest results ofeach of the plurality of subjects; d) a software module configured toassess a similarity between the first test result and the plurality ofretest results of each of the plurality of subjects by ranking thedistances in a non-descending order; e) a software module configured toassess a rank precision for the first subject based on the rank ofdistances from the first test result to each of the plurality of retestresults of the first subject; f) a software module configured to repeatb), c), d), and e) for each of the plurality of subjects; and g) asoftware module configured to evaluate the test-retest precision basedon the rank precision for each of the plurality of subjects.

In yet another aspect, disclosed herein are non-transitorycomputer-readable storage media encoded with a computer programincluding instructions executable by a processor to create anapplication for evaluating test-retest precision of a test and a retestof a plurality of subjects using fractional rank precision (FRP) ormean-average precision (MAP), comprising: a) a database, in a computermemory, of a test result and a plurality of retest results from each ofthe plurality of subjects, wherein the test result and the plurality ofretest results are described in one or more feature spaces, and one ofthe test result and the plurality of retest results is collected from avision test machine; b) a software module configured to select a firstsubject from the plurality of subjects and a first test result of thefirst subject; c) a software module configured to calculate distancesfrom the first test result to each of the plurality of retest results ofeach of the plurality of subjects; d) a software module configured toassess a similarity between the first test result and the plurality ofretest results of each of the plurality of subjects by ranking thedistances in a non-descending order; e) a software module configured toassess a rank precision for the first subject based on the rank ofdistances from the first test result to each of the plurality of retestresults of the first subject; f) a software module configured to repeatb), c), d), and e) for each of the plurality of subjects; and g) asoftware module configured to evaluate the test-retest precision basedon the rank precision for each of the plurality of subjects.

In yet another aspect, disclose herein are computer-implemented methodsof evaluating test-retest precision of a test and a retest of aplurality of subjects using fractional rank precision (FRP) ormean-average precision (MAP), comprising: a) collecting, by a computer,a test result and a retest result of each of the plurality of subjects,wherein the test result and the retest result are described in one ormore feature spaces and one of the test result and the retest result iscollected from a vision test machine; b) selecting, by the computer, anth subject from the plurality of subjects and the test result of thenth subject; c) calculating, by the computer, distances from the testresult of the nth subject to the retest result of each of the pluralityof subjects; d) assessing, by the computer, a similarity between thetest result of the nth subject and the retest result of each of theplurality of subjects by ranking the distances in a non-descendingorder; e) assessing, by the computer, a rank precision for the nthsubject based on the rank of the distance from the test result of thenth subject to the retest result of the nth subject; f) repeating, bythe computer, steps b), c), d), and e) for 1≤n≤N, N being a total numberof subjects in the plurality of subjects; and g) evaluating, by thecomputer, the test-retest precision based on the rank precision for eachof the plurality of subjects.

In yet another aspect, disclosed herein are computer-implemented systemscomprising a digital processing device comprising at least oneprocessor, an operating system configured to perform executableinstructions, a memory, and a computer program including instructionsexecutable by the digital processing device to create an application ofevaluating test-retest precision of a test and a retest of a pluralityof subjects using fractional rank precision (FRP) or mean-averageprecision (MAP), comprising: a) a software module configured to collecta test result and a retest result from each of the plurality ofsubjects, wherein the test result and the retest result are described inone or more feature spaces and one of the test result and the retestresult is collected from a vision test machine; b) a software moduleconfigured to select a nth subject from the plurality of subjects and atest result of the nth subject; c) a software module configured tocalculate distances from the test result of the nth subject to theretest result of each of the plurality of subjects; d) a software moduleconfigured to assess a similarity between the test result of the nthsubject and the retest result of each of the plurality of subjects byranking the distances in a non-descending order; e) a software moduleconfigured to assess a rank precision for the nth subject based on arank of a distance from the test result of the nth subject to the retestresult of the nth subject; f) repeating, by the computer, steps b), c),d), and e) for 1≤n≤N, N being a total number of subjects in theplurality of subjects; and g) a software module configured to evaluatethe test-retest precision based on the rank precision for each of theplurality of subjects.

In yet another aspect, disclosed herein are computer-implemented methodsof evaluating test-retest precision of a test and a retest of aplurality of subjects using fractional rank precision (FRP) ormean-average precision (MAP), comprising: a) collecting, by a computer,a test result and a plurality of retest results from each of theplurality of subjects, wherein the test result and the plurality ofretest results are described in one or more feature spaces, and one ofthe test result and the plurality of retest results is collected from avision test machine; b) selecting, by the computer, a nth subject fromthe plurality of subjects and a test result of the nth subject; c)calculating, by the computer, distances from the test result of the nthsubject to each of the plurality of retest results of each of theplurality of subjects; d) assessing, by the computer, a similaritybetween the test result of the nth subject and the plurality of retestresults of each of the plurality of subjects by ranking the distances ina non-descending order; e) assessing, by the computer, a rank precisionfor the first subject based on the rank of distances from the testresult of the nth subject to each of the plurality of retest results ofthe first subject; f) repeating, by the computer, steps b), c), d), ande) for 1≤n≤N, N being a total number of subjects in the plurality ofsubjects; and g) evaluating, by the computer, the test-retest precisionbased on the rank precision for each of the plurality of subjects.

In yet another aspect disclosed herein are computer-implemented systemscomprising a digital processing device comprising at least oneprocessor, an operating system configured to perform executableinstructions, a memory, and a computer program including instructionsexecutable by the digital processing device to create an application ofevaluating test-retest precision of a test and a retest of a pluralityof subjects using fractional rank precision (FRP) or mean-averageprecision (MAP), comprising: a) a software module configured to collecta test result and a retest result from each of the plurality ofsubjects, wherein the test result and the retest result are described inone or more feature spaces and one of the test result and the retestresult is collected from a vision test machine; b) a software moduleconfigured to select a nth subject from the plurality of subjects and atest result of the nth subject; c) a software module configured tocalculate distances from the test result of the nth subject to theretest result of each of the plurality of subjects; d) a software moduleconfigured to assess a similarity between the test result of the nthsubject and the retest result of each of the plurality of subjects byranking the distances in a non-descending order; e) a software moduleconfigured to assess a rank precision for the nth subject based on arank of a distance from the test result of the nth subject to the retestresult of the nth subject; f) repeating, by the computer, steps b), c),d), and e) for 1≤n≤N, N being a total number of subjects in theplurality of subjects; and g) a software module configured to evaluatethe test-retest precision based on the rank precision for each of theplurality of subjects.

In yet another aspect, disclosed herein are computer-implemented systemscomprising a digital processing device comprising at least oneprocessor, an operating system configured to perform executableinstructions, a memory, and a computer program including instructionsexecutable by the digital processing device to create an application ofevaluating test-retest precision of a test and a retest of a pluralityof subjects using fractional rank precision (FRP) or mean-averageprecision (MAP), comprising: a) a software module configured to collecta test result and a plurality of retest results from each of theplurality of subjects, wherein the test result and the plurality ofretest results are described in one or more feature spaces, and one ofthe test result and the plurality of retest results is collected from avision test machine; b) a software module configured to select a nthsubject from the plurality of subjects and a test result of the nthsubject; c) a software module configured to calculate distances from thetest result of the nth subject to each of the plurality of retestresults of each of the plurality of subjects; d) a software moduleconfigured to assess a similarity between the test result of the nthsubject and the plurality of retest results of each of the plurality ofsubjects by ranking the distances in a non-descending order; e) asoftware module configured to assess a rank precision for the nthsubject based on the rank of distances from the test result of the nthsubject to each of the plurality of retest results of the nth subject;f) a software module configured to repeat b), c), d), and e) for 1≤n≤N,N being a total number of subjects in the plurality of subjects; and g)a software module configured to evaluate the test-retest precision basedon the rank precision for each of the plurality of subjects.

Overview

Human vision is one of the most important senses for many everydaytasks, and blindness ranks highly among most-feared ailments. Therefore,early diagnosis and treatment of vision loss are critical. As a result,regular monitoring of visual function at least in at-risk populationswould be desirable. Certain vision tests rest on computationallyintensive algorithms to provide a more comprehensive description ofvisual function. Thus, they have the potential to improve clinical careand research by more precise measurement of the effects of diseaseprogression or ophthalmic interventions. A common proxy to studyprecision of a test is to assess its test-retest reliability; however,standard methods to assess clinical precision and test-retestreliability may be inadequate or even misleading for the more complex,higher-dimensional test outputs. Two major sources of imprecisiontypically are noise in the measurement device and moment-to-momentvariability of the physiological phenomenon under observation. Toestimate the contribution of noise in the device, precision is oftenassessed by the reliability of repeated measurements. The standard toolsfor this assessment are the intra-class correlation coefficient (ICC)and the Bland-Altman coefficient of repeatability (CoR). However, it isimportant to note that repeatability measures are an indirect assessmentonly; in the extreme case, a test with a binary outcome, as anon-limiting example, light perception, may have almost perfectreliability but little discriminatory power for most of the population.Furthermore, the ICC is dominated by the values at either end of thetest range and is therefore sensitive to outliers. The Bland-Altmancoefficient of repeatability, which is defined as 1.96 times thestandard deviation of differences between repeated measurements, doesnot suffer from this problem, and also may provide an intuitivethreshold for how much change between two tests should be consideredstatistically significant. However, this threshold rests on theassumption that tests are homoscedastic, as a non-limiting example, themeasurement error is independent of the magnitude of the ground truth(e.g. patients with poor vision perform tests as reliably asnormal-sighted controls); its usefulness is limited by quantization ofmany tests, in addition to the evidence of heteroscedasticity in visiontesting data. Moreover, absolute CoR values do not directly relate toclinical meaningfulness, and do not allow comparing the reliability ofdifferent tests with outputs of different magnitude or differentdimensionality.

In some embodiments, the methods and systems disclosed herein relate tothe information retrieval field. In some embodiments, the methods andsystems disclosed herein use method(s) in information retrieval. Themethods and systems disclosed herein, in various embodiments, areapplied in order to evaluate similarity, repeatability, reliability,precision, or variability of a test-retest pair. In some embodiments,the methods and systems disclosed herein are used for multiple subjectswith each subject having at least one test and at least one retestresult. In some embodiments, method disclosed herein, for non-limitingexamples, MAP and FRP, are particularly useful for test-retest resultsin higher-dimensional feature spaces including quick CSF (qCSF) results,traditional CSF measurements at different spatial frequencies, imagingmodalities such as fundus photography or OCT, and perimetry results ofdifferent locations in the visual field. For traditional tests withscalar outcomes, it is relatively easy to statistically describe someaspects of test-retest variability, for example “95% of test-retestpairs will be less than X units apart.” However, there are limitationswith that method (as a non-limiting example, dependence on test rangeand quantization); such limitations get much worse in amulti-dimensional space, when test-retest differences are expressed as amulti-dimensional vector. However, MAP or FRP methods are suited fordescribing relationships or point distances within higher-dimensionalfeature spaces. In some embodiments, MAP and FRP methods can not only beapplied to individual features or combinations of features of a specificvision test, but also be applied to combine results from different testsincluding one or more functional, behavior, or structural tests.Further, MAP or FRP methods can be used to compare and select individualfeatures or different combination of features based on their test-retestprecision performance. In some embodiments, CoR and ICC are morevulnerable to artifacts than MAP and FRP for measuring test-retestsimilarity. In some embodiments, MAP and FRP are more accurate incapturing a test's precision by expressing test-retest variability interms of population variability. In some embodiments, MAP and FRPpenalize coarse quantization in the test-retest results.

Tests and Retests

In some embodiments, the methods and systems disclosed herein include atest and a retest. In some embodiments, a test and a retest isexamination of one or more visionary feature using specialized andsuitable medical machine capable of measuring the feature(s). In someembodiments, a test and a retest is examination of one or more visionaryfeatures as disclosed herein. In some embodiments, a test and a retestis a procedure whose result is described by one or more visionaryfeatures as disclosed herein. In some embodiments, a test and a retestis an examination or procedure using one or more specialized machine asdisclosed herein. In some embodiment, a test or a retest is a same testwith all testing conditions substantially identical for the test and theretest except the time of testing. In further embodiments, the testingconditions include one or more selected from: the testing subjects, thetesting instruments, the testing procedure, the testing method, theresult recording method, the testing time, the testing artifacts, andthe testing noise level. In some cases, the testing conditions includeone or more selected from: a test processing method, a test evaluationmethod and a test scoring method. In some cases, a test and a retest isperformed with only one or more differences in testing conditions. As anonlimiting example, the only substantial difference between testingconditions in a test and a retest may be the testing time. In someembodiments, a test or a retest is a same test with an identical testingprotocol except the time of testing. In some cases, a test and a retestis performed with one or more differences in testing conditions. As anonlimiting example, a test and a retest is performed before and afterat least one therapeutic intervention, respectively, and also withdifferent testing time. In some embodiments, a test and a retest isperformed before and after at least some disease progression has takenplace, respectively. In some embodiments, a test and a retest is adifferent test.

In some cases, a test or a retest as disclosed herein may be anysuitable vision tests or medical tests that can be described in afeature space. As nonlimiting examples, a test or a retest may be one ormore selected from: an acuity test, an acuity threshold test, an acuityslope test, a sensitivity test, a perimetry test, a color sensitivitytest, a spatial frequency sensitivity test with at least one spatialfrequency ranging from 0 cycles per degree of visual angle (cpd) to 50cpd, a contrast sensitivity test, a contrast sensitivity function (CSF)test, a CSF test with at least one spatial frequency ranging from 0 to50 cpd, a peak spatial frequency test, a peak sensitivity test, asensitivity bandwidth test, an optical coherence tomography (OCT) test,a ultrasound test, a magnetic resonance imaging (MRI) test, an X-raytest, a microscopic test, a computerized adaptive contrast sensitivitytest using a computerized contrast sensitivity testing device, a qCSFtest using a qCSF testing device, or any other imaging modality. In someembodiments, a test or a retest examines at least one preselectedparameter of at least a vision disease. In some embodiments, a test or aretest examines at least one preselected parameter of at least onetherapeutic treatment or intervention.

In some cases, the test may be the first test being performed on thesubject(s). Alternatively, the test may be a test preceding a number ofretests. Alternatively, a test may be any test that proceeds or followsa certain number of retests. In some cases, each subject may have one ormore retests. In further cases, the number of retests for each subjectmay or may not be identical. As a nonlimiting example, four humansubjects may each have a test; three of the four subjects may have 3retests each; the last subject may have 2 retests. The total number ofsubjects is 4, and the total number of retests may be 11. A test and aretest may be in the same feature space. Alternatively, a test and aretest may be in two different feature spaces.

In some cases, a test (retest) may not be limited to vision test. Insome cases, a test (retest) may be any suitable test of physiologicalfunctions or pathological functions. In some cases, a test (retest) maybe any suitable tests that may be described in a feature space.

Results

In some embodiments, at least one test and at least one retest isperformed with a same subject. In some embodiments, a test and a retestincludes at least one test result and a retest result. In someembodiments, a test and a retest includes at least a number of testresults and a number of retest results, the number being any real numberno smaller than 1. In some embodiments, a test result or a retest resultis indicative of visual performance of a subject. In some embodiments, atest result or a retest result is indicative of at least one diseaseprogression. In some embodiments, a test result or a retest result isindicative of effect of at least one therapeutic intervention.

In some embodiments, a test result or a retest result includes at leastone test score. In some embodiments, a test (retest) score is a scalar.In further embodiments, a test (retest) result includes a score ranginganywhere from 0 to 1. In some embodiments, a test (retest) score isnormalized by the maximum score or a pre-selected number to be withinthe range of 0 to 1. In some embodiments, a test (retest) score is anynon-negative real number. In some embodiments, a test (retest) score isone or more real numbers. In some embodiments, a test (retest) score ismulti-dimensional. In some embodiments, a test (retest) score is avector. In some embodiments, a test (retest) score is amulti-dimensional vector. In some embodiments, the number of dimensionsof a test (retest) score is determined by or equal to the number ofdimensions of a feature space. In some embodiments, a test (retest)score includes at least one scalar score for each dimension in thefeature space that a test and a retest was performed and/or measured.

In some embodiments, a test (retest) score in at least one dimension isweighted. In some embodiments, a test (retest) score or the raw data ofa test or a retest is processed in at least one dimension. In someembodiments, the processing of a test (retest) score includes one ormore selected from weighting, normalization, noise reduction, filtering,translation, quantization, and rounding up.

In some cases, a test or a retest as disclosed herein may be anysuitable vision tests or medical tests that may be conducted onspecialized medical machinery other than a generic computer. In someembodiments, a test (retest) result may be collected from a specializedmachine that enables measurement of visionary parameters of humansubjects. In some embodiments, the methods and systems, and digitalprocessing devices as disclosed herein include a connection to one ormore specialized machinery that enables measurement of visionaryparameters of human subjects. In some embodiments, the specializedmachinery disclosed herein may include but are not limited to: acomputerized adaptive contrast sensitivity testing device, a qCSFtesting device, an OCT machine, a Mill machine, a X-ray machine, anultrasound machine, a visual field testing machine, a fundus photographysystem, a dark adaptation measurement machine, an auto-refractormachine, a frequency-doubling technology machine, a tonometer machine,an aberrometer machine, an eye-tracking device, an Adaptive SensoryTechnology Sentio device (“Next-generation vision testing: the quickCSF” in Current Directions in Biomedical Engineering 2015; 1:131-134 byDon et. al.), and an ocular alignment machine. In some embodiments, thespecialized vision test machine may not include a vision acuity chart.In some cases, there may be a wired or a wireless connection between thespecialized machinery used to collect test (retest) results. In someembodiments, the wired or wireless connection may be direct. In othercases, the connection may be indirect, and the collected result(s) ofraw data may be preprocessed at another machine using preprocessingsteps as disclosed herein. In some embodiments, the methods and systemsdisclosed herein may include a processor or a digital processing devicefor preprocessing of raw data collected from a one or more specializedmachines.

In some cases, the methods and systems as disclosed herein receive testresults from the specialized machinery via one or more connectionselected from: a network, an Ethernet, an Internet, a cable, a phoneline, a non-transitory computer-readable media, an online cloud, adatabase, a phone line, an aux connection, a Bluetooth connection, a auxconnection, or the like.

Test-Retest Precision

In some embodiments, test-retest similarity, test-retest precision,test-retest reliability, and test-retest repeatability areinterchangeable or equivalent as disclosed herein. In some embodiments,test-retest similarity, test-retest precision, test-retest reliability,and test-retest repeatability indicate how similar the test-retestresults are. In some cases, the test-retest reliability helps determineif the difference in test-retest is of medical significance or not. Insome cases, the test-retest reliability helps determine if thedifference in test-retest is caused by testing noise, testing errors,and or any other factors unrelated to medical conditions or therapeuticinterventions.

In some embodiments, the test-retest precision is a measure of howsimilar a test-retest(s) pair is. In some embodiments, the test-retestprecision is an assessment of the similarity of test-retest results fromone subject. In some embodiments, the test-retest precision is anassessment of the similarity of test-retest results from more than onesubject, with the test-retest precision assessed for each of thesubjects so each subject is the “one subject” once in a test-retestprecision measurement. Each test-retest pair may include one test andone retest for the same subject. Each test-retest pair may include onetest and one retest for two different subjects.

Each test-retest pair may include one test and more than one retest ofthe same subjects. Each test-retest pair may include one test result andmore than one retest results of the same subjects.

In some embodiments, CoR or the Bland-Altman CoR is calculated as

${{CoR} = {1.96\sqrt{\frac{1}{N - 1}{\sum\limits_{n = 1}^{N}\left\lbrack {x_{n,2} - x_{n,1} - {\frac{1}{N}{\sum\limits_{m = 1}^{N}x_{m,2}}} - x_{m,1}} \right\rbrack^{2}}}}},$wherein each subject x_(n) has a test-retest pair, x_(n,1) and x_(n,2),and the total number of subject is N.

In some embodiments, ICC is calculated as

${ICC} = {\frac{2}{{\sum\limits_{n = 1}^{N}\left( {x_{n,1} - \overset{\_}{x}} \right)^{2}} + {\sum\limits_{n = 1}^{N}\left( {x_{n,2} - \overset{\_}{x}} \right)^{2}}}{\sum\limits_{n = 1}^{N}{\left( {x_{n,1} - \overset{\_}{x}} \right)\left( {x_{n,2} - \overset{\_}{x}} \right)}}}$wherein each subject x_(n) has a test-retest pair, x_(n,1) and x_(n,2),the total number of subject is N, and whereinx=1/(2N)Σ_(n=1) ^(N) x _(n,1) +x _(n,2).Vision-Based Features

In some embodiment, a vision based feature is any feature that may betested or examined using a vision-related testing device. In someembodiments, non-limiting examples of a feature include but is notlimited to one or more selected from: a retinal layer thickness, aretinal blood vessel thickness, a retinal haemorrhage index, astructural descriptor of an image of the eye, a cup-to-disc ratio, adrusen count, an edema count, a frequency doubling technology-relatedfeature, a perimetry subfield threshold, a visual field loss summarystatistic, a visual field sensitivity map, an interocular pressure, asaccadic latency, a smooth pursuit gain index, a dark-adaptation rodintercept time, a chromatic contrast threshold, and a temporal frequencythreshold. In some embodiments, non-limiting examples of featuresinclude one or more selected from: a parameter of acuity, an acuitythreshold, an acuity slope, a sensitivity parameter, a sensitivityparameter of CSF, a CSF distribution, a probability distribution of CSF,the peak spatial frequency fmax of CSF, peak sensitivity ymax of CSF,bandwidth β of CSF, and a low-frequency truncation parameter δ of CSF, asensitivity parameter of quick CSF (qCSF), qCSF, a qCSF distribution, ajoint distribution of qCSF over four parameters (a distribution ofresults in a four-dimensional space, with each result describing a CSFcurve and each curve living in a two-dimensional space of spatialfrequency and contrast), an area under the Log CSF (AULCSF) integratedover the spatial frequency range from at least 1.5 to 18 cycles perdegree (cpd), a CSF acuity, as a non-limiting example, the spatialfrequency point where the CSF described by four parameters reaches acontrast threshold of 100%, a sensitivity threshold of the CSF that isdescribed by four parameters for at least one spatial frequency rangingfrom 1 to 50 cpd, and the spatial point for which the CSF reaches apre-determined contrast threshold.

In some embodiments, a contrast threshold is the inverse of sensitivity.In some embodiments, the four parameters of CSF or qCSF are a peakspatial frequency fmax of CSF, peak sensitivity γmax of CSF, bandwidth βof CSF, and a low-frequency truncation parameter δ of CSF. In someembodiments, the at least one spatial frequency ranging from 1 to 18 cpdincludes at least 1 cpd, 1.5 cpd, 3 cpd, 6 cpd, 12 cpd, and 18 cpd. Insome embodiments, AULCSF represents the median, average, or processedAULCSF computed over spatial frequencies ranging from 1.5 cpd to 18 cpd.In some embodiments, the CSF is a curve in a two dimensional space ofspatial frequency and contrast.

In further embodiments, a feature includes at least one transformationor an operation of at least one vision feature. In some embodiments, atransformation or an operation includes one or more selected from: amean, a median, a standard deviation, a multiplication by a pre-selectednumber or another feature, a square, a cube, a nth order multiplication,a square root, a nth root, a division by a pre-selected number, a randomsubsampling, a Fourier Transform, a rounding, and a change inquantization level.

In some embodiments, a feature includes a probability or statisticaldistribution of at least one feature. In some embodiments, a featureincludes a joint probability or statistical distribution of at least twofeatures. In some embodiments, a feature includes information derivedfrom a probability distribution, a statistical distribution of onefeature, a joint probability distribution of at least two features, ajoint statistical distribution of at least two features. In someembodiments, a feature includes a scalar feature.

In some embodiments, the methods and systems disclosed herein includes afeature space. In some embodiments, a feature space includes anycombination of the above-mentioned features and/or transformations,where each feature corresponds to one dimension in a multi-dimensionalspace. In some embodiments, the feature space includes a dimension sizeno less than one. In some embodiments, at least a number of dimensionsin the feature space are orthogonal to each other, the number being noless than 2.

In some embodiments, the feature space undergoes another transformationthat scales and rotates the coordinate system of the feature space. As anon-limiting example, individual dimension (feature) is scaled withdifferent factors to make the range of feature values comparable. Insome embodiments, the coordinate system is rotated by a whitening stepthat de-correlates the features. In some embodiments, a principlecomponent analysis (PCA) is applied that de-correlates the features andreduces the dimensionality of the feature space to preserve only theprinciple components of the feature space. In some embodiments, a testresult is represented in a feature space as one multi-dimensionalvector, or as a distribution of multi-dimensional vectors.

In some embodiments, in the visual domain, the predominant tool forfunctional assessment is acuity, which measures the smallest size of astimulus (typically, a letter) that an observer can recognize at fullcontrast. In some embodiments, while acuity is well established and canbe very useful e.g. for adjusting optical correction, it has at leasttwo issues in domains where precision is paramount, such as in clinicaltrials. First, the variability of repeated measurements makes the methodinsensitive to subtle changes in vision, e.g. due to disease progressionor treatment. Second, some ophthalmic and neurologic conditions affectacuity only moderately, despite of effects on visual function. In someembodiments, the contrast sensitivity function (CSF) relates anobserver's ability to recognize a spatial pattern not only to its size,but also to its contrast. In some embodiments, comparing to acuity, theCSF correlates better with performance in visually guided everydayactivities, for non-limiting examples, driving, walking, and the abilityto recognize faces.

Traditional tests such as paper charts typically only return one number(e.g. “20/20 vision”, i.e. 100%). In some embodiments, for more complextests such as the qCSF, which returns joint probability distributionsover four parameters (a distribution of test results in afour-dimensional space, with each result describing a CSF curve, each ofthese curves living in a two-dimensional space of spatial frequency andcontrast), an arbitrary number of features is generated based on thesedistributions to create a feature space of arbitrary dimensionality; forexample, the median AULCSF is a one-dimensional feature space, and thecombination of median AULCSF and median CSF acuity would give atwo-dimensional feature space.

In some embodiments, visual sensitivity is comprehensively described byCSF, but current routine clinical care does not include its assessmentbecause of the time-consuming need to estimate thresholds for a largenumber of spatial frequencies. In some embodiments, the quick CSF (qCSF)method dramatically reduces testing times by using a Bayesianinformation maximization rule. In some embodiments, the qCSF method usesa computationally intensive algorithm and optimizes stimulus selectionby computing the expected information gain over a very large set ofpossible stimuli and the probability distribution of possible CSFs,given the history of previous trials. In some embodiments, the CSF is adescription of visual function that assigns a contrast sensitivity (themost subtle gray/gray difference that can be discerned) to everypossible spatial frequency (size of the stimulus), thus, the CSF spaceis two-dimensional (contrast and spatial frequency). People havemeasured the full CSF before, but the combinatorial complexity of theadditional dimension made those tests impractical for clinical care—theytook 30-60 minutes; the method developed by Lesmes and colleagues,however, reduced testing times to 2-3 minutes (Lesmes et. al. “Bayesianadaptive estimation of the contrast sensitivity function: The quick CSFmethod,” Journal of Vision 2010 10(3) hence the ‘quick’ CSF).

In some embodiments, the quantization level may be changed by changingthe number of digits in a test and a retest score. In other embodiments,the quantization level is not changed by changing the number of digitsbut may be determined by the number of different test scores that canresult from the test. As an example, a test with three scores, ⅓, 1/7,and 1, even with infinite number of digits the test itself provides onlythree scores thus only very quantized outcomes. In some embodiments, thenumber of digits is the number of fractional digits. In someembodiments, at least two to at least 5 intermediate steps are includedbetween two adjacent digits after the decimal point. In some cases, as anon-limiting example, a quantization level with 5 fractional digits isless coarse and less quantized than a quantization level with 4fractional digits given that the fraction digits are for an identicaldigit after the decimal point. In some embodiments, as a non-limitingexample, with 5 fractional steps for the second digit after the decimalpoint, 1.028 is rounded to 1.02 and 1.071 is rounded to 1.08).

Similarities

The methods and systems disclosed herein, in various embodiments, areapplied in order to evaluate similarity, repeatability, reliability,precision, or variability of a test-retest pair. In some cases, thesimilarity, repeatability, reliability, precision, or variability of atest-retest pair is equivalent or interchangeable as disclosed herein.

In some cases, the similarity of a test-retest pair is assessed byranking distances between the selected test result and each retestresult of the plurality of subjects and obtaining a rank of the distancefrom the selected test result to the retest result of the same subject.As a nonlimiting example in FIG. 1, a similarity of test-retest pair ofsubject 2 (left) is lower than the similarity of test-retest pair ofsubject 3 (right), as the distance of test-retest pair of subject 2(left) ranks 3 out of 4 while the distance of test-retest pair ofsubject 3 (right) ranks 1 out of 4. A rank precision may be obtainedbased on the similarities. In this case, the rank precision for subject2 using MAP is ⅓ and the rank precision for subject 3 using MAP is 1.

Ranks

In some embodiments, the method described herein includes a rank. Insome embodiments, a rank is given to each retest result. Further, eachretest result may include one or more retests for each of the pluralityof subjects participated in the test-retest precision measurement. Insome embodiments, a rank is a real number ranging from 0 to N−1, whereinN is the total number of subjects, the total number of retests, thetotal number of tests, or the total number of tests and retests. In someembodiments, a rank is a real number ranging from 1 to N, wherein N isthe total number of subjects, the total number of retests, the totalnumber of tests, or the total number of tests and retests. In someembodiments, the rank is given to each retest according to a distance ofeach retest to at least one specified test. As a nonlimiting example, atotal of 6 subjects each have one test and two retests results. A rankmay be given to each of the 12 retest results based on their distance tothe test of the first subject.

Other example of ranks may be seen in FIG. 1. Referring to FIG. 1, in aparticular embodiment, four subjects are measured in a test-retest eventand test-retest precision is evaluated. In this embodiment, test andretest scores for each of the four subjects are plotted, and thedistance of each retest result to the test result of subject 2 (leftpanel) and subject 3 (right panel) is obtained. Correspondingly, in thesame embodiment, the rank of distance is optionally calculated. In thisembodiment, for subject 2, retest score has a rank of 3 in a totalnumber of 4 subjects, so that the average precision of subject 2'sretest is the inverse of rank of the retest, thus, the precision is ⅓.In the same embodiment, similarly, for subject 3, retest score has arank of 1 in a total number of 4 subjects, so that the average precisionof subject 3's retest is the inverse of rank of the retest, thus, theprecision is 1.

In some embodiments, the rank number is reversibly related to thedistance value. As a non-limiting example, the retest result with theshortest distance to at least one specified test result has a rank of 0;the retest result with the longest distance to at least one specifiedtest result has a rank of N−1. In some embodiments, the retest resultsand/or subjects of the retest results are sorted in a list with thecorresponding distance in a monotonically non-descending order: theretest result and/or subject at the top of the sorted list has a rank of0; the retest result and/or subject at the bottom of the sorted list hasa rank of N−1. In some embodiments, the rank is normalized by the totalnumber of subjects. In some embodiments, the rank is weighted givenpreselected weighting for at least one rank value.

In some embodiments, a rank is any number ranging from 1 to N, wherein Nis the total number of subjects, the total number of retest, or thetotal number of tests. In some embodiments, the rank number isreversibly related to the distance value. As a non-limiting example, theretest result with the shortest distance to at least one specified testresult has a rank of 1; the retest result with the longest distance toat least one specified test result has a rank of N. In some embodiments,the retest results and/or subjects of the retest results are sorted in alist with the corresponding distance in a monotonically non-descendingorder: the retest result and/or subject at the top of the sorted listhas a rank of 1; the retest result and/or subject at the bottom of thesorted list has a rank of N.

In some embodiments, the rank is the same for at least two retestresults if the distance is identical for at least two retest resultsand/or subjects. In some embodiments, the identical rank is calculatedas an average of adjacent ranks which would be given to the at least tworetest results if they were not identical without affecting the ranks ofother retest results. In some embodiments, as a non-limiting example,distances of 0.1, 0.1, 0.2, and 0.3 get ranked as 1.5, 1.5, 3, and 4,respectively.

Mean Average Precision (MAP) and Fractional Rank Precision (FRP)

The methods and systems disclosed herein may include one or moresuitable test-retest precision measurement methods. In some embodiments,the methods and systems disclosed herein include a MAP or a FRPcalculation for the evaluation of test-retest precision, test-retestrepeatability, test-retest reliability, or test-retest variation. Insome cases, the methods and systems disclosed herein may include acombination of MAP and FRP calculation, the weighting of MAP or FRPcalculation may be preselected based on empirical experience,simulation, trails, or any other suitable factors.

In some cases, fractional rank precision may not be limited tonon-integer ranking to break ties. In some cases, fractional ranking mayalso include normalized ranking precision or normalized ranking. In someembodiments, fractional ranking is equivalent to and interchangeablewith normalized ranking. In some embodiments, fractional rankingprecision is equivalent to and interchangeable with normalized rankprecision (NRP).

In some embodiments, with one test and one retest for each subject, rankprecision is directly calculated as (1−rank(Ri)/N), wherein the rankgoes from 0 to N−1, wherein N is the total number of subjects, andwherein i is the selected subject.

In some embodiments, the FRP is the mean of the rank precision for eachsubject of all the subjects in a test-retest event.

In some embodiments, with at least one test and at least one retest foreach subject, the average precision (AveP) is directly calculated as1/rank (Ri), wherein rank goes from 1 to N, wherein N is the totalnumber of subjects, and i is the selected subject. The average precisionmay be interchangeable or equivalent to the ranks precision in variousembodiments as disclosed herein. In some embodiment, each subject hasmore than one test results and more than one retest result, and the AvePis the average of 1/rank(Ri) for each retest of the selected subject,wherein i is the selected subject. In some embodiments, the MAP is themean of AveP for each subject of all the subjects in a test-retestevent.

In some embodiments, the MAP is calculated as

${{MAP} = {\frac{1}{N}{\sum\limits_{n = 1}^{N}{{Precision}_{x_{n,1}}\left( x_{n,2} \right)}}}},$wherein each subject x_(n) has a test-retest pair, x_(n,1) and x_(n,2),the total number of subjects is N, and wherein precision of a testresult x_(n,1) is based on the rank of its corresponding retest x_(n,2)as

${{Precision}_{x_{{n,1}\;}}\left( x_{n,2} \right)} = {\frac{1}{{rank}_{x_{n,1}}\left( x_{n,2} \right)}.}$

In some embodiments, the FRP is calculated as

${{FRP} = {1 - {\frac{1}{N}{\sum\limits_{n = 1}^{N}\frac{{{rank}_{x_{n,1}}\left( x_{n,2} \right)} - 1}{N}}}}},$wherein each subject x_(n) has a test-retest pair, x_(n,1) and x_(n,2),the total number of subjects is N.

In some embodiments, the methods and systems disclosed herein includeone or more recursive steps. In further cases, one or more recursivesteps repeats only once for each of the plurality of subjects. Infurther cases, one or more recursive steps repeats for more than oncefor each of the plurality of subjects when each subject has more thanone retest results, the repetition number being equal to the number ofretests for individual subjects.

Distances

In some embodiments, the distance between test-retest results isrepresented by a number, or a multi-dimensional vector. In someembodiments, the distance may be any distance calculated using anysuitable calculation methods for two scalars or vectors representing thetest-retest results pair in one or two feature spaces. In someembodiments, a distance is one or more selected from: the Euclideandistance, the Mahalanobis distance, and the Manhattan distance. In someembodiments, the Manhattan distance represents a sum of dimension-wiseabsolute differences.

In some embodiments, the distance between test results is represented byone or more selected from: the distributions of multi-dimensionalvectors, the mean, the median, or other quantiles of the distribution ofEuclidean or Manhattan distances between the vectors. In someembodiments, the distance between test-retest results is represented bythe mean, median, or any other quantiles of the distribution ofMahalanobis distances between sampled points of the one distribution andthe whole other distribution. In some embodiments, the Mahalanobisdistance expresses distance in terms of the width of the data ellipsoidin the multi-dimensional feature space.

Referring to FIG. 1, in a particular embodiment, four subjects aremeasured in a test-retest event and test-retest precision is evaluatedusing MAP. In this embodiment, test and retest scores for each of thefour subjects are plotted, and the distance of each retest result to thetest result of subject 2 (left panel) and subject 3 (right panel) isobtained. Correspondingly, in the same embodiment, the rank of distanceis optionally calculated. In this embodiment, MAP for all four subjectsis optionally calculated as the mean of average precision for each ofthe four subjects. In this case, MAP of all four subjects is 0.708.

Referring to FIG. 2, in a particular embodiment, effect of quantizationof test and retest scores on different of test-retest precision (TRP)measures is examined. In this embodiment, a median AULCSF computed overspatial frequencies from 1.5 cpd to 18 cpd is measured using differenttest-retest precision measures. In this embodiment, the quantizationlevel affects accuracy in test-retest precision measures including CoR,ICC, FRP, and MAP (FIG. 2). In this embodiment, MAP precision score,which reflects measurement precision using MAP, gradually decreasesmonotonically with increased quantization. Additionally, in the sameembodiment, coarse quantization that renders a test uninformative maylead to ‘perfect’ reliability as measured by CoR and ICC. For example,when quantization level is less than 1, ICC may show an inaccuratelyhigh precision score. In this embodiment, for intermediate quantizationlevel of about 0.10 to about 1.00, MAP is more sensitive and accuratethan ICC.

Referring to FIG. 3, in a particular embodiment, boxplots of thedistributions of FRP are plotted over the subjects for various visionaryfeatures. The mean of the distribution represents the TRP value forvarious visionary features. In this embodiment, using FRP measure,test-retest precision may be different when it is based on differentvision-based feature. AULCSF may be slightly more accurate with a highaverage precision while 1.0 cpd may have a lower test-retest precision.

Referring to FIG. 4, in a particular embodiment, effect of quantizationon different test-retest precision measures is examined. In thisembodiment, a median AULCSF computed over spatial frequencies from 1.5cpd to 18 cpd is measured using different test-retest precision (TRP)measures. These different measures include CoR, ICC, MAP, and FRP. Inthis embodiment, CoR is particularly vulnerable to quantizationdifferences and shows non-monotonic behavior with increased level ofquantization. In the same embodiment, the ICC also behavesnon-monotonically with increased level of quantization. In the sameembodiment, the change of FRP evaluation of precision stays monotonic asthe level of quantization increases. In this embodiment, thequantization level affects the accuracy in test-retest precisionmeasures including CoR, ICC, MAP, and FRP.

Referring to FIG. 7, in a particular embodiment, the effect of choosinga different prior over the CSF search space on FRP, is determined as afunction of trial number. In this case, both the uniform and thepopulation prior lead to very similar FRP results over the first fewtrials, as only very little information about the true CSF is available.In the same embodiments, for the time range between 10 to 30 trials, theanalysis initialized with the population prior shows better convergence.In the same embodiment, for the time range after 30 trials, thedifference between FRP diminishes as both approaches converge; however,there is still a small benefit for population after 50 trials (FRP of0.871 and 0.864, respectively).

Digital Processing Device

In some embodiments, the platforms, systems, media, and methodsdescribed herein include a digital processing device, or use of thesame. In further embodiments, the digital processing device includes oneor more hardware central processing units (CPUs) or general purposegraphics processing units (GPGPUs) that carry out the device'sfunctions. In still further embodiments, the digital processing devicefurther comprises an operating system configured to perform executableinstructions. In some embodiments, the digital processing device isoptionally connected a computer network. In further embodiments, thedigital processing device is optionally connected to the Internet suchthat it accesses the World Wide Web. In still further embodiments, thedigital processing device is optionally connected to a cloud computinginfrastructure. In other embodiments, the digital processing device isoptionally connected to an intranet. In other embodiments, the digitalprocessing device is optionally connected to a data storage device.

In accordance with the description herein, suitable digital processingdevices include, by way of non-limiting examples, server computers,desktop computers, laptop computers, notebook computers, sub-notebookcomputers, netbook computers, netpad computers, set-top computers, mediastreaming devices, handheld computers, Internet appliances, mobilesmartphones, tablet computers, personal digital assistants, video gameconsoles, and vehicles. Those of skill in the art will recognize thatmany smartphones are suitable for use in the system described herein.Those of skill in the art will also recognize that select televisions,video players, and digital music players with optional computer networkconnectivity are suitable for use in the system described herein.Suitable tablet computers include those with booklet, slate, andconvertible configurations, known to those of skill in the art.

In some embodiments, the digital processing device includes an operatingsystem configured to perform executable instructions. The operatingsystem is, for example, software, including programs and data, whichmanages the device's hardware and provides services for execution ofapplications. Those of skill in the art will recognize that suitableserver operating systems include, by way of non-limiting examples,FreeBSD, OpenBSD, NetBSD®, Linux, Apple® Mac OS X Server®, Oracle®Solaris®, Windows Server®, and Novell® NetWare®. Those of skill in theart will recognize that suitable personal computer operating systemsinclude, by way of non-limiting examples, Microsoft® Windows®, Apple®Mac OS X®, UNIX®, and UNIX-like operating systems such as GNU/Linux®. Insome embodiments, the operating system is provided by cloud computing.Those of skill in the art will also recognize that suitable mobile smartphone operating systems include, by way of non-limiting examples, Nokia®Symbian® OS, Apple® iOS®, Research In Motion® BlackBerry OS®, Google®Android®, Microsoft® Windows Phone® OS, Microsoft® Windows Mobile® OS,Linux®, and Palm® WebOS®. Those of skill in the art will also recognizethat suitable media streaming device operating systems include, by wayof non-limiting examples, Apple TV®, Roku®, Boxee®, Google TV®, GoogleChromecast®, Amazon Fire®, and Samsung® HomeSync®. Those of skill in theart will also recognize that suitable video game console operatingsystems include, by way of non-limiting examples, Sony® PS3®, Sony®PS4®, Microsoft® Xbox 360°, Microsoft Xbox One, Nintendo® Wii®,Nintendo® Wii U®, and Ouya®.

In some embodiments, the device includes a storage and/or memory device.The storage and/or memory device is one or more physical apparatusesused to store data or programs on a temporary or permanent basis. Insome embodiments, the device is volatile memory and requires power tomaintain stored information. In some embodiments, the device isnon-volatile memory and retains stored information when the digitalprocessing device is not powered. In further embodiments, thenon-volatile memory comprises flash memory. In some embodiments, thenon-volatile memory comprises dynamic random-access memory (DRAM). Insome embodiments, the non-volatile memory comprises ferroelectric randomaccess memory (FRAM). In some embodiments, the non-volatile memorycomprises phase-change random access memory (PRAM). In otherembodiments, the device is a storage device including, by way ofnon-limiting examples, CD-ROMs, DVDs, flash memory devices, magneticdisk drives, magnetic tapes drives, optical disk drives, and cloudcomputing based storage. In further embodiments, the storage and/ormemory device is a combination of devices such as those disclosedherein.

In some embodiments, the digital processing device includes a display tosend visual information to a user. In some embodiments, the display is acathode ray tube (CRT). In some embodiments, the display is a liquidcrystal display (LCD). In further embodiments, the display is a thinfilm transistor liquid crystal display (TFT-LCD). In some embodiments,the display is an organic light emitting diode (OLED) display. Invarious further embodiments, on OLED display is a passive-matrix OLED(PMOLED) or active-matrix OLED (AMOLED) display. In some embodiments,the display is a plasma display. In other embodiments, the display is avideo projector. In still further embodiments, the display is acombination of devices such as those disclosed herein.

In some embodiments, the digital processing device includes an inputdevice to receive information from a user. In some embodiments, theinput device is a keyboard. In some embodiments, the input device is apointing device including, by way of non-limiting examples, a mouse,trackball, track pad, joystick, game controller, or stylus. In someembodiments, the input device is a touch screen or a multi-touch screen.In other embodiments, the input device is a microphone to capture voiceor other sound input. In other embodiments, the input device is a videocamera or other sensor to capture motion or visual input. In furtherembodiments, the input device is a Kinect, Leap Motion, or the like. Instill further embodiments, the input device is a combination of devicessuch as those disclosed herein.

Referring to FIG. 8, in a particular embodiment, an exemplary digitalprocessing device 101 is programmed or otherwise configured to measuretest-retest reliability using FRP, MAP or other suitable precisionevaluation methods. The device 101 can regulate various aspects oftest-retest precision measurement of the present disclosure, such as,for example, formulating test-retest reliability as an informationretrieval problem, and ranking retest measurements by their distance toa subject's test measurement. As another example, it may assess asimilarity between a test result and a retest result. In thisembodiment, the digital processing device 101 includes a centralprocessing unit (CPU, also “processor” and “computer processor” herein)105, which can be a single core or multi core processor, or a pluralityof processors for parallel processing. The digital processing device 101also includes memory or memory location 110 (e.g., random-access memory,read-only memory, flash memory), electronic storage unit 115 (e.g., harddisk), communication interface 120 (e.g., network adapter) forcommunicating with one or more other systems, and peripheral devices125, such as cache, other memory, data storage and/or electronic displayadapters. The memory 110, storage unit 115, interface 120 and peripheraldevices 125 are in communication with the CPU 105 through acommunication bus (solid lines), such as a motherboard. The storage unit115 can be a data storage unit (or data repository) for storing data.The digital processing device 101 can be operatively coupled to acomputer network (“network”) 130 with the aid of the communicationinterface 120. The network 130 can be the Internet, an internet and/orextranet, or an intranet and/or extranet that is in communication withthe Internet. The network 130 in some cases is a telecommunicationand/or data network. The network 130 can include one or more computerservers, which can enable distributed computing, such as cloudcomputing. The network 130, in some cases with the aid of the device101, can implement a peer-to-peer network, which may enable devicescoupled to the device 101 to behave as a client or a server. The digitalprocessing device 101 can be operatively connected to one or morespecialized medical device (not shown) via the network 130. Suchconnection may enable data collection from the medical device; the datamay include one or more test results, retest results, and other relatedtest and subject information. The specialized medical device isconfigured to measure visionary features(s) of one or more subjects.

Continuing to refer to FIG. 8, the CPU 105 can execute a sequence ofmachine-readable instructions, which can be embodied in a program orsoftware. The instructions may be stored in a memory location, such asthe memory 110. The instructions can be directed to the CPU 105, whichcan subsequently program or otherwise configure the CPU 105 to implementmethods of the present disclosure. Examples of operations performed bythe CPU 105 can include fetch, decode, execute, and write back. The CPU105 can be part of a circuit, such as an integrated circuit. One or moreother components of the device 101 can be included in the circuit. Insome cases, the circuit is an application specific integrated circuit(ASIC) or a field programmable gate array (FPGA).

Continuing to refer to FIG. 8, the storage unit 115 can store files,such as drivers, libraries and saved programs. The storage unit 115 canstore user data, e.g., user preferences and user programs. The digitalprocessing device 101 in some cases can include one or more additionaldata storage units that are external, such as located on a remote serverthat is in communication through an intranet or the Internet.

Continuing to refer to FIG. 8, the digital processing device 101 cancommunicate with one or more remote computer systems through the network130. For instance, the device 101 can communicate with a remote computersystem of a user. Examples of remote computer systems include personalcomputers (e.g., portable PC), slate or tablet PCs (e.g., Apple® iPad,Samsung® Galaxy Tab), telephones, Smart phones (e.g., Apple® iPhone,Android-enabled device, Blackberry®), or personal digital assistants.

Methods as described herein can be implemented by way of machine (e.g.,computer processor) executable code stored on an electronic storagelocation of the digital processing device 101, such as, for example, onthe memory 110 or electronic storage unit 115. The machine executable ormachine readable code can be provided in the form of software. Duringuse, the code can be executed by the processor 105. In some cases, thecode can be retrieved from the storage unit 115 and stored on the memory110 for ready access by the processor 105. In some situations, theelectronic storage unit 115 can be precluded, and machine-executableinstructions are stored on memory 110.

Non-Transitory Computer Readable Storage Medium

In some embodiments, the platforms, systems, media, and methodsdisclosed herein include one or more non-transitory computer readablestorage media encoded with a program including instructions executableby the operating system of an optionally networked digital processingdevice. In further embodiments, a computer readable storage medium is atangible component of a digital processing device. In still furtherembodiments, a computer readable storage medium is optionally removablefrom a digital processing device. In some embodiments, a computerreadable storage medium includes, by way of non-limiting examples,CD-ROMs, DVDs, flash memory devices, solid state memory, magnetic diskdrives, magnetic tape drives, optical disk drives, cloud computingsystems and services, and the like. In some cases, the program andinstructions are permanently, substantially permanently,semi-permanently, or non-transitorily encoded on the media.

Computer Program

In some embodiments, the platforms, systems, media, and methodsdisclosed herein include at least one computer program, or use of thesame. A computer program includes a sequence of instructions, executablein the digital processing device's CPU, written to perform a specifiedtask. Computer readable instructions may be implemented as programmodules, such as functions, objects, Application Programming Interfaces(APIs), data structures, and the like, that perform particular tasks orimplement particular abstract data types. In light of the disclosureprovided herein, those of skill in the art will recognize that acomputer program may be written in various versions of variouslanguages.

The functionality of the computer readable instructions may be combinedor distributed as desired in various environments. In some embodiments,a computer program comprises one sequence of instructions. In someembodiments, a computer program comprises a plurality of sequences ofinstructions. In some embodiments, a computer program is provided fromone location. In other embodiments, a computer program is provided froma plurality of locations. In various embodiments, a computer programincludes one or more software modules. In various embodiments, acomputer program includes, in part or in whole, one or more webapplications, one or more mobile applications, one or more standaloneapplications, one or more web browser plug-ins, extensions, add-ins, oradd-ons, or combinations thereof.

Web Application

In some embodiments, a computer program includes a web application. Inlight of the disclosure provided herein, those of skill in the art willrecognize that a web application, in various embodiments, utilizes oneor more software frameworks and one or more database systems. In someembodiments, a web application is created upon a software framework suchas Microsoft® .NET or Ruby on Rails (RoR). In some embodiments, a webapplication utilizes one or more database systems including, by way ofnon-limiting examples, relational, non-relational, object oriented,associative, and XML database systems. In further embodiments, suitablerelational database systems include, by way of non-limiting examples,Microsoft® SQL Server, mySQL™, and Oracle®. Those of skill in the artwill also recognize that a web application, in various embodiments, iswritten in one or more versions of one or more languages. A webapplication may be written in one or more markup languages, presentationdefinition languages, client-side scripting languages, server-sidecoding languages, database query languages, or combinations thereof. Insome embodiments, a web application is written to some extent in amarkup language such as Hypertext Markup Language (HTML), ExtensibleHypertext Markup Language (XHTML), or eXtensible Markup Language (XML).In some embodiments, a web application is written to some extent in apresentation definition language such as Cascading Style Sheets (CSS).In some embodiments, a web application is written to some extent in aclient-side scripting language such as Asynchronous Javascript and XML(AJAX), Flash® Actionscript, Javascript, or Silverlight®. In someembodiments, a web application is written to some extent in aserver-side coding language such as Active Server Pages (ASP),ColdFusion®, Perl, Java™, JavaServer Pages (JSP), Hypertext Preprocessor(PHP), Python™, Ruby, Tcl, Smalltalk, WebDNA®, or Groovy. In someembodiments, a web application is written to some extent in a databasequery language such as Structured Query Language (SQL). In someembodiments, a web application integrates enterprise server productssuch as IBM® Lotus Domino®. In some embodiments, a web applicationincludes a media player element. In various further embodiments, a mediaplayer element utilizes one or more of many suitable multimediatechnologies including, by way of non-limiting examples, Adobe® Flash®,HTML 5, Apple® QuickTime®, Microsoft® Silverlight®, Java™, and Unity®.

Mobile Application

In some embodiments, a computer program includes a mobile applicationprovided to a mobile digital processing device. In some embodiments, themobile application is provided to a mobile digital processing device atthe time it is manufactured. In other embodiments, the mobileapplication is provided to a mobile digital processing device via thecomputer network described herein.

In view of the disclosure provided herein, a mobile application iscreated by techniques known to those of skill in the art using hardware,languages, and development environments known to the art. Those of skillin the art will recognize that mobile applications are written inseveral languages. Suitable programming languages include, by way ofnon-limiting examples, C, C++, C #, Objective-C, Java™, Javascript,Pascal, Object Pascal, Python™, Ruby, VB.NET, WML, and XHTML/HTML withor without CSS, or combinations thereof.

Suitable mobile application development environments are available fromseveral sources. Commercially available development environmentsinclude, by way of non-limiting examples, AirplaySDK, alcheMo,Appcelerator®, Celsius, Bedrock, Flash Lite, .NET Compact Framework,Rhomobile, and WorkLight Mobile Platform. Other development environmentsare available without cost including, by way of non-limiting examples,Lazarus, MobiFlex, MoSync, and Phonegap. Also, mobile devicemanufacturers distribute software developer kits including, by way ofnon-limiting examples, iPhone and iPad (iOS) SDK, Android™ SDK,BlackBerry® SDK, BREW SDK, Palm® OS SDK, Symbian SDK, webOS SDK, andWindows® Mobile SDK.

Those of skill in the art will recognize that several commercial forumsare available for distribution of mobile applications including, by wayof non-limiting examples, Apple® App Store, Google® Play, Chrome WebStore, BlackBerry® App World, App Store for Palm devices, App Catalogfor webOS, Windows® Marketplace for Mobile, Ovi Store for Nokia®devices, Samsung® Apps, and Nintendo® DSi Shop.

Standalone Application

In some embodiments, a computer program includes a standaloneapplication, which is a program that is run as an independent computerprocess, not an add-on to an existing process, e.g., not a plug-in.Those of skill in the art will recognize that standalone applicationsare often compiled. A compiler is a computer program(s) that transformssource code written in a programming language into binary object codesuch as assembly language or machine code. Suitable compiled programminglanguages include, by way of non-limiting examples, C, C++, Objective-C,COBOL, Delphi, Eiffel, Java™, Lisp, Python™, Visual Basic, and VB .NET,or combinations thereof. Compilation is often performed, at least inpart, to create an executable program. In some embodiments, a computerprogram includes one or more executable complied applications.

Web Browser Plug-In

In some embodiments, the computer program includes a web browser plug-in(e.g., extension, etc.). In computing, a plug-in is one or more softwarecomponents that add specific functionality to a larger softwareapplication. Makers of software applications support plug-ins to enablethird-party developers to create abilities which extend an application,to support easily adding new features, and to reduce the size of anapplication. When supported, plug-ins enable customizing thefunctionality of a software application. For example, plug-ins arecommonly used in web browsers to play video, generate interactivity,scan for viruses, and display particular file types. Those of skill inthe art will be familiar with several web browser plug-ins including,Adobe® Flash® Player, Microsoft® Silverlight®, and Apple® QuickTime®. Insome embodiments, the toolbar comprises one or more web browserextensions, add-ins, or add-ons. In some embodiments, the toolbarcomprises one or more explorer bars, tool bands, or desk bands.

In view of the disclosure provided herein, those of skill in the artwill recognize that several plug-in frameworks are available that enabledevelopment of plug-ins in various programming languages, including, byway of non-limiting examples, C++, Delphi, Java™ PHP, Python™, and VB.NET, or combinations thereof.

Web browsers (also called Internet browsers) are software applications,designed for use with network-connected digital processing devices, forretrieving, presenting, and traversing information resources on theWorld Wide Web. Suitable web browsers include, by way of non-limitingexamples, Microsoft® Internet Explorer®, Mozilla® Firefox, Google®Chrome, Apple® Safari®, Opera Software® Opera®, and KDE Konqueror. Insome embodiments, the web browser is a mobile web browser. Mobile webbrowsers (also called mircrobrowsers, mini-browsers, and wirelessbrowsers) are designed for use on mobile digital processing devicesincluding, by way of non-limiting examples, handheld computers, tabletcomputers, netbook computers, subnotebook computers, smartphones, musicplayers, personal digital assistants (PDAs), and handheld video gamesystems. Suitable mobile web browsers include, by way of non-limitingexamples, Google® Android® browser, RIM BlackBerry® Browser, Apple®Safari®, Palm® Blazer, Palm® WebOS® Browser, Mozilla® Firefox® formobile, Microsoft® Internet Explorer® Mobile, Amazon® Kindle® Basic Web,Nokia® Browser, Opera Software® Opera® Mobile, and Sony® PSP™ browser.

Software Modules

In some embodiments, the platforms, systems, media, and methodsdisclosed herein include software, server, and/or database modules, oruse of the same. In view of the disclosure provided herein, softwaremodules are created by techniques known to those of skill in the artusing machines, software, and languages known to the art. The softwaremodules disclosed herein are implemented in a multitude of ways. Invarious embodiments, a software module comprises a file, a section ofcode, a programming object, a programming structure, or combinationsthereof. In further various embodiments, a software module comprises aplurality of files, a plurality of sections of code, a plurality ofprogramming objects, a plurality of programming structures, orcombinations thereof. In various embodiments, the one or more softwaremodules comprise, by way of non-limiting examples, a web application, amobile application, and a standalone application. In some embodiments,software modules are in one computer program or application. In otherembodiments, software modules are in more than one computer program orapplication. In some embodiments, software modules are hosted on onemachine. In other embodiments, software modules are hosted on more thanone machine. In further embodiments, software modules are hosted oncloud computing platforms. In some embodiments, software modules arehosted on one or more machines in one location. In other embodiments,software modules are hosted on one or more machines in more than onelocation.

Databases

In some embodiments, the platforms, systems, media, and methodsdisclosed herein include one or more databases, or use of the same. Inview of the disclosure provided herein, those of skill in the art willrecognize that many databases are suitable for storage and retrieval ofinformation including one or more selected from: one or more ofsubjects, one or more test types, one or more retest types, one or moretest results associated with each subject, one or more retestsassociated with each subject, and one or more features for thetest/retest results. In various embodiments, suitable databases include,by way of non-limiting examples, relational databases, non-relationaldatabases, object oriented databases, object databases,entity-relationship model databases, associative databases, and XMLdatabases. Further non-limiting examples include SQL, PostgreSQL, MySQL,Oracle, DB2, and Sybase. In some embodiments, a database isinternet-based. In further embodiments, a database is web-based. Instill further embodiments, a database is cloud computing-based. In otherembodiments, a database is based on one or more local computer storagedevices.

Incorporation by Reference

All publications, patents, and patent applications mentioned in thisspecification are herein incorporated by reference to the same extent asif each individual publication, patent, or patent application wasspecifically and individually indicated to be incorporated by reference.

Certain Definitions

Unless otherwise defined, all technical terms used herein have the samemeaning as commonly understood by one of ordinary skill in the art towhich this invention belongs. As used in this specification and theappended claims, the singular forms “a,” “an,” and “the” include pluralreferences unless the context clearly dictates otherwise. Any referenceto “or” herein is intended to encompass “and/or” unless otherwisestated.

EXAMPLES

The following illustrative examples are representative of embodiments ofthe software applications, systems, and methods described herein and arenot meant to be limiting in any way.

Example 1

Comfortably seated, subjects holding an iPad 4 that is set to a meanscreen luminance of 185 cd/m2 at a viewing distance of 60 cm. In eachvision test-retest trial, a randomly chosen bandpass-filtered Sloanletter is presented for 500 ms, and the subject then had to registertheir response by touch-selecting a letter from an array on the screen(10-Alternatives Forced Choice). The stimulus space is log-spaced andcomprised 24 spatial scales (0.64 to 41 cycles per degree (cpd)) and 48contrast levels (0.2 to 100%). In order to avoid uninformative regionsof the stimulus space, the next stimulus was then randomly chosen fromthe top 10 percent of the distribution of expected information gain. Allsubjects run the test multiple times. Controls are tested monocularly inboth eyes and binocularly; to estimate reliability of the procedure, oneof these three conditions (randomly chosen) is repeated. Myopes aretested both with and without their optical correction, with one repeat,for a total of 7 conditions. We report only on the repeated data.

Two traditional methods, ICC and CoR are compared with the FRP methoddescribed herein for their measurements of the test-retest precision ofdifferent features for vision sensitivity testing. Such visionsensitivity testing features include AULCSF, which is the area under thelog CSF integrated between spatial frequencies from 1.5 to 18 cpd, theCSF acuity, and the sensitivity threshold of CSF at six differentspatial frequencies including 1 cpd, 1.5 cpd, 3 cpd, 6 cpd, 12 cpd, and18 cpd. Results of comparison are shown in Table 1. As shown in FIG. 3,boxplots of the distributions of FRP are plotted over a plurality ofsubjects for various visionary features that may be included in afeature space. These various features are non-limiting examples ofvision-based features that may be used to measure qualitative orquantitative aspects of human vision. The mean of the distributionrepresents the TRP value for each feature. FRP method disclosed hereinindicates that AULCSF most precisely identifies test-retest pairs eventhough it summarizes over a broad range of spatial frequencies.Additionally, the sensitivity at 6 cpd has a much higher CoR, 0.293,than the CSF acuity feature, 0.193, but more precisely identifiestest-retest pairs according to the FRP method disclosed herein, 0.848vs. 0.844. The ICC is vulnerable to outliers: adding a single subjectwith very poor vision (for non-limiting example, test-retest sensitivityat 1.0 cpd of 0.1 and 0.05 respectively) substantially changes ICC from0.868 to 0.817; while FRP is much less affected by outliers, and thechange is less than 0.005.

Bland-Altman CoR and ICC for two selected features are graphically shownin FIG. 5.

TABLE 1 Feature ICC CoR FRP AULCSF 0.980 0.225 0.871 CSF acuity 0.9670.193 0.844 1 cpd 0.368 0.203 0.752 1.5 cpd 0.945 0.196 0.793 3 cpd0.969 0.265 0.827 6 cpd 0.978 0.293 0.848 12 cpd 0.963 0.310 0.839 18cpd 0.917 0.369 0.823

Referring to FIG. 5, in a particular embodiment, test-retest precisionis evaluated for test-retest scored using features including AULCSF andsensitivity of CSF at 1.0 cpd. In this embodiment, test-retest scoresare compared using traditional Bland-Altman plot (top panels) andcorrelation plot (bottom panels). In this embodiment, dashed linesrepresent bias and 95% confidence intervals for test-retest differences.In this embodiment, the Bland-Altman plots do not show obviousdependencies of test-retest differences on the magnitude of the meantest score. The standard deviation of differences for 1.0 cpd isslightly smaller than for AULCSF. In the same embodiment, scattercorrelation plots in the bottom panels also show strong agreementbetween test and retest scores.

Referring to Table 1, in a particular embodiment, fractional rankprecision is calculated for different vision sensitivity featuresincluding AULCSF, which is the area under the log CSF between spatialfrequencies from 1.5 to 18 cpd, the CSF acuity, and the sensitivitythreshold of CSF at six different spatial frequencies including 1 cpd,1.5 cpd, 3 cpd, 6 cpd, 12 cpd, and 18 cpd. In this embodiment, theAULCSF shows the largest area under the precision-recall curve, thus,the greatest fractional rank precision among all the examined features.In this embodiment, the AULCSF has the smallest test-retest variabilityand greatest test-retest precision.

Example 2

Fractional rank precision values for different features of qCSF areexamined in test-retest trials with multiple subjects. As shown in FIG.6, with dashed lines indicating chance level (0.5) and FRP forsubjective prior assessment (0.767). Among scalar features, the AULCSFyields higher FRP levels than CSF acuity (after 50 trials, FRP of 0.871and 0.844, respectively) as it summarizes over a large range of spatialfrequencies. Multi-dimensional features, however, identify test-retestpairs with greater precision. The combination of high-frequency cutoff(CSF acuity) and the sensitivity near the presumed peak of the CSF (1.5cpd) already yields a FRP of 0.885; further adding sensitivity at amid-range spatial frequency (6 cpd) improves FRP to 0.899, and theaddition of the AULCSF results in the highest FRP of 0.901. Notably, FRPseems not to have plateaued after 50 trials for the multi-dimensionalfeatures indicating gain in precision is obtainable by running the qCSFmethod for more trials.

Example 3

Comfortably seated, subjects holding an iPad 4 that is set to a meanscreen luminance of 185 cd/m2 at a viewing distance of 60 cm. In eachvision test-retest trial, a randomly chosen bandpass-filtered Sloanletter is presented for 500 ms, and the subject then had to registertheir response by touch-selecting a letter from an array on the screen(10-Alternatives Forced Choice). The stimulus space is log-spaced andcomprised 24 spatial scales (0.64 to 41 cycles per degree (cpd)) and 48contrast levels (0.2 to 100%). In order to avoid uninformative regionsof the stimulus space, the next stimulus was then randomly chosen fromthe top 10 percent of the distribution of expected information. Allsubjects run the test multiple times. Controls are tested monocularly inboth eyes and binocularly; to estimate reliability of the procedure, oneof these three conditions (randomly chosen) is repeated. Myopes aretested both with and without their optical correction, with one repeat,for a total of 7 conditions. We report only on the repeated data.

Two traditional methods, ICC and CoR are compared with the MAP methoddescribed herein for their measurements of the test-retest precision ofdifferent features for vision sensitivity testing. Such visionsensitivity testing features include AULCSF, which is the area under thelog CSF between spatial frequencies from 1.5 to 18 cpd, the CSF acuity,and the sensitivity threshold of CSF at six different spatialfrequencies including 1 cpd, 1.5 cpd, 3 cpd, 6 cpd, 12 cpd, and 18 cpd.Results of comparison are shown in Table 2. Among all the testedfeatures, AULCSF yields highest MAP. In other words, AULCSF is the bestfeature to identify a pair of test-retest measurements, even though itsummarizes over a broad range of spatial frequencies. Sensitivities forlow spatial frequencies including 1 cpd and 1.5 cpd vary less acrosssubjects, therefore are less discriminative across subjects.

TABLE 2 Feature ICC CoR MAP AULCSF 0.980 0.225 0.206 CSF acuity 0.9670.193 0.200 1 cpd 0.868 0.203 0.147 1.5 cpd 0.945 0.196 0.185 3 cpd0.969 0.265 0.164 6 cpd 0.975 0.293 0.149 12 cpd 0.963 0.310 0.109 18cpd 0.917 0.369 0.110

While preferred embodiments of the present invention have been shown anddescribed herein, it will be obvious to those skilled in the art thatsuch embodiments are provided by way of example only. Numerousvariations, changes, and substitutions will now occur to those skilledin the art without departing from the invention. It should be understoodthat various alternatives to the embodiments, of the invention describedherein may be employed in practicing the invention. It is intended thatthe following claims define the scope of the invention and that methodsand structures within the scope of these claims and their equivalents becovered thereby.

What is claimed is:
 1. A system for evaluating test-retest precisionusing fractional rank precision (FRP) or mean-average precision (MAP),comprising: a vision test machine; one or more processors; and acomputer-readable storage device coupled to the one or more processorsand having instructions stored thereon which, when executed by the oneor more processors, cause the one or more processors to performoperations comprising: receiving a test result of a test for one subjectof a plurality of subjects and a retest result of the test for each ofthe subjects, where the test result and each of the retest resultscomprises one or more feature spaces, and wherein one of the test resultand the retest results is received from the vision test machine;calculating distances from the test result to the retest results;assessing a similarity between the test result and the retest results byranking the distances in a non-descending order; assessing a rankprecision for the subject based on a rank of a distance from the testresult to the retest result of the subject; and evaluating a test-retestprecision of the test based on the rank precision for the subject. 2.The system of claim 1, wherein the test comprises a first vision test.3. The system of claim 2, wherein the retest result for the subjectcomprises a repeat of the first vision test after a therapy or treatmentat a different time point.
 4. The system of claim 2, wherein the firstvision test comprises a vision acuity test, a contrast sensitivityfunction (CSF) test, or an optical coherence tomography (OCT) test. 5.The system of claim 1, wherein the one or more feature spaces areone-dimensional.
 6. The system of claim 1, wherein the one or morefeature spaces are multi-dimensional.
 7. The system of claim 1, whereina feature of the feature space comprises: a median area under log ofcontrast sensitivity function (AULCSF) computed over the spatialfrequency range of 1.5 to 6 cycles per degree (cpd), a median AULCSFcomputed over the spatial frequency range of 6 to 12 cpd, a medianAULCSF computed over the spatial frequency range of 12 to 18 cpd, amedian AULCSF computed over the spatial frequency range of
 1. 5 to 18cpd, a contrast sensitivity function (CSF) acuity, a parameter of CSF, acontrast sensitivity for at least one spatial frequency selected from 1,1.5, 3, 6, 12, and 18 cpd, a peak sensitivity of the CSF, or a spatialfrequency at which a CSF reaches a pre-determined contrast threshold. 8.The system of claim 1, wherein the rank comprises a real number rangingfrom 0 to N−1, wherein N represents a total number of subjects in theplurality of subjects.
 9. The system of claim 1, wherein the distancecomprises: a Euclidean distance, a Manhattan distance, or a Mahalanobisdistance.
 10. The system of claim 1, wherein the operations forassessing the rank precision for the subject comprise: calculating anormalized rank, wherein the normalized rank comprises the rank of thedistance divided by a total number of subjects of the plurality ofsubjects; and calculating the rank precision, wherein the rank precisionis equal to one subtracted by the normalized rank.
 11. The system ofclaim 1, wherein the rank precision comprises an inverse of the rank ofthe retest result of the subject.
 12. The system of claim 1, wherein theoperations comprise: assessing a rank precision for each of the othersubjects based on a received test result for the other subjects, andwherein the test-retest precision comprises a mean of the rank precisionfor each of the plurality of subjects.
 13. The system of claim 1,wherein the rank comprises a real number ranging from 1 to N, wherein Nrepresent a total number of subjects of the plurality of subjects. 14.The system of claim 1, wherein the vision test machine comprise: acomputerized adaptive contrast sensitivity testing device, a quickcontrast sensitivity function (qCSF) testing device, an opticalcoherence tomography (OCT) machine, a magnetic resonance imaging (MRI)machine, an ultrasound machine, a visual field testing machine, a fundusphotography system, a dark adaptation measurement machine, anauto-refractor machine, a frequency-doubling threshold machine, atonometer machine, an aberrometer machine, an eye-tracking device, or anocular alignment machine.
 15. A computer-implemented method forevaluating test-retest precision using fractional rank precision (FRP)or mean-average precision (MAP), the method being executed by one ormore processors and comprising: receiving a test result of a test forone subject of a plurality of subjects and a retest result of the testfor each of the subjects, where the test result and each of the retestresults comprises one or more feature spaces, and wherein one of thetest result and the retest results is received from a vision testmachine; calculating distances from the test result to the retestresults; assessing a similarity between the test result and the retestresults by ranking the distances in a non-descending order; assessing arank precision for the subject based on a rank of a distance from thetest result to the retest result of the subject; and evaluating atest-retest precision of the test based on the rank precision for thesubject.
 16. The method of claim 15, wherein the test comprises a firstvision test, wherein the retest result for the subject comprises arepeat of the first vision test after a therapy or treatment at adifferent time point, and wherein the first vision test comprises avision acuity test, a contrast sensitivity function (CSF) test, or anoptical coherence tomography (OCT) test.
 17. The method of claim 15,wherein a feature of the feature space comprises: a median area underlog of contrast sensitivity function (AULCSF) computed over the spatialfrequency range of 1.5 to 6 cycles per degree (cpd), a median AULCSFcomputed over the spatial frequency range of 6 to 12 cpd, a medianAULCSF computed over the spatial frequency range of 12 to 18 cpd, amedian AULCSF computed over the spatial frequency range of 1.5 to 18cpd, a contrast sensitivity function (CSF) acuity, a parameter of CSF, acontrast sensitivity for at least one spatial frequency selected from 1,1.5, 3, 6, 12, and 18 cpd, a peak sensitivity of the CSF, or a spatialfrequency at which a CSF reaches a pre-determined contrast threshold.18. The method of claim 15, wherein assessing the rank precision for thesubject comprises: calculating a normalized rank, wherein the normalizedrank comprises the rank of the distance divided by a total number ofsubjects of the plurality of subjects; and calculating the rankprecision, wherein the rank precision is equal to one subtracted by thenormalized rank.
 19. The method of claim 15, comprising: assessing arank precision for each of the other subjects based on a received testresult for the other subjects, and wherein the test-retest precisioncomprises a mean of the rank precision for each of the plurality ofsubjects.
 20. One or more non-transitory computer-readable storage mediacoupled to one or more processors and having instructions stored thereonwhich, when executed by the one or more processors, cause the one ormore processors to perform operations comprising: receiving a testresult of a test for one subject of a plurality of subjects and a retestresult of the test for each of the subjects, where the test result andeach of the retest results comprises one or more feature spaces, andwherein one of the test result and the retest results is received from avision test machine; calculating distances from the test result to theretest results; assessing a similarity between the test result and theretest results by ranking the distances in a non-descending order;assessing a rank precision for the subject based on a rank of a distancefrom the test result to the retest result of the subject; and evaluatinga test-retest precision of the test based on the rank precision for thesubject.